DIAGNOSING J2EE PERFORMANCE PROBLEMS

T H R O U G H O U T T H E A P P L I C AT I O N L I F E C Y C L E
Ta b l e o f C o n t e n t s

Challenges for J2EE Applications Diagnosing Performance Problems

and Performance 3 with Mercury for J2EE 6

Typical J2EE Application Summary 8

Performance Problems 4

Alternative Technologies for Capturing

J2EE Performance Data 5

ABSTRACT
MANY LARGE-SCALE AND COMPLEX ENTERPRISE APPLICATIONS ARE NOW BUILT AND DEPLOYED USING THE J2EE

ARCHITECTURE. HOWEVER, MANY OF THESE APPLICATIONS SUFFER FROM POOR PERFORMANCE AND SCALABILITY BECAUSE

THE FOCUS OF THE DEVELOPMENT PROCESS IS ON FUNCTIONALITY, WHILE PERFORMANCE AND SCALABILITY ARE DEALT

WITH AS AN AFTERTHOUGHT.

THIS PAPER WILL PRESENT TECHNIQUES FOR DELIVERING HIGH PERFORMANCE APPLICATIONS TO PRODUCTION, MANAGING

AND MEASURING THE PERFORMANCE OF APPLICATIONS, AND DIAGNOSING THE TOUGHEST J2EE PROBLEMS THROUGHOUT

THE ENTIRE APPLICATION LIFECYCLE. THE PAPER WILL EXAMINE THE VARIOUS TYPES OF PERFORMANCE ISSUES THAT NEED

TO BE DEALT WITH AT EACH STAGE OF THE LIFECYCLE AND WHAT DIFFERENT DIAGNOSTIC TOOLS AND TECHNIQUES CAN

BEST RESOLVE THEM.

2
CHALLENGES FOR J2EE APPLICATIONS
AND PERFORMANCE
Today’s enterprises are choosing to build business
applications that leverage the power, portability,
and rapid development of Java technology.
J2EE offers many advantages to developers, but
introduces new challenges for the development,
performance diagnosis, tuning, deployment, and
management of applications. Successful delivery
and management of typically complex J2EE-based
applications requires evaluating performance
throughout the entire application lifecycle.
An application may perform well in the develop-
ment and QA environment, but fail to scale or
may exhibit transient performance problems
in production. It is important to understand
the impact of the infrastructure in which the
application runs and the behavior of the many
application components as they interact under
load. From the diagnostic perspective, it is
important to be able to isolate the problem by
tier of the application architecture, by application
component, and to have progressive drill-down
visibility into J2EE performance problems, the
J2EE environment, and into the actual code,
with sufficient detail to determine the root
cause of the problems.
There are additional factors that can increase
the difficulties of application delivery and
management. The deployment lifecycle for many
Web-facing J2EE applications is compressed,
due to increased pressures for quick time to
market. Boundaries between development, QA,
deployment, and production stages and IT
groups are blurred. Centralized IT organizations
may be managing hundreds of applications,
with little depth of each. IT staff skills for
J2EE may not be developed enough.
Many applications have not been sufficiently
architected for performance and scalability, with
thorough consideration of design and usage
patterns, and adequate attention to planning
and testing performance against well defined
service objectives. J2EE scalability capabilities,
while extensive, do not substitute for such
efforts. Later in the lifecycle, applications may
be pushed into production to meet deadlines,
with insufficient validation of performance or
scalability and inadequate tools to see into
application internal behavior, making it difficult
and expensive to fix problems. Getting to the
root of performance problems in the complex,
distributed, and dynamic J2EE environment is
truly a challenge.
Performance Evaluation and Diagnostic
Needs Throughout the Application Lifecycle
During the application lifecycle, there are many
• During staging or pre-production, deployment
IT stakeholders in application performance. They
include application architects, developers, load
testers and engineers in QA, application support
staff, and site operations in production. They
share common requirements for performance
diagnostics, but also have specific roles and
needs unique to their environments.
• In the design and specification phases, a
topic outside the scope of this diagnostic
paper, performance and scalability issues
need to be considered, incorporated, and
• In live production, operations and application
specific objectives set.
• In the development and unit-testing phase,
profiling tools can be useful to validate
performance, along with functionality.
Developers should test the performance of
components against identified latency and
system resource utilization criteria.
3
• During quality assurance cycles, load testing
typically follows integrated functional and
regression testing. A complete application,
including all interfaces with external systems,
should be fully load tested prior to software
release. Objectives include scalability and
capacity estimation under load that realistically
represents expected live usage, along with
visibility into the internal performance behavior
of the application and actionable data on
bottlenecks. This should include transaction
breakdown of latencies for each J2EE tier
and method, along with additional specific
root cause diagnostic information.
Profiling tools used at the developer desktop
cannot be used in QA during load testing for
performance diagnosis, due to the high over-
head they impose. J2EE diagnostic tools used
at this stage need to be designed for load and
should be integrated with load testing tools to
boost productivity and testing effectiveness.
teams should determine the performance and
scalability of the application in the specific
IT environment and configuration that will
serve live users, verify projected peak load
capability, and tune the application and
infrastructure to meet its designed transaction
throughput and response time service levels.
Load testing and performance diagnostic
tools that can optimize both application
and infrastructure, and drill down to solve
problems, are essential during this stage.
managers need the ability to continuously
monitor the health of the entire system,
measure the performance of the application,
and, if something starts to go wrong, be able
to quickly alert, triage, isolate, and pinpoint
the root cause of the problem. Needs include
real-time monitors that provide drill-down
visibility into the application, and in-depth,
offline analytic tools for application support to
dig into deep details of performance problems.
It is sometimes difficult to reproduce tough
production problems in staging or QA test beds.
A toolset that provides drill-down from a broad
user and business transaction view all the way to
detailed internal diagnostics can be essential for
eliminating persistent or recurring performance
problems. The toolset should provide a combi-
nation of agentless infrastructure monitoring,
low-overhead J2EE monitoring agents, and the
ability to narrowly target problem areas for deep
measurements. Such toolsets should be able
to diagnose the toughest problems, such as
intermittent slow methods, full transaction
tracing including arguments, memory leaks,
synchronization, and deadlock issues.
Throughout the entire application lifecycle,
J2EE-specific visibility and diagnostic capability
should be integrated with and complementary to
multi-platform, multi-protocol tools for functional
testing, load testing, and application performance
management. Common tools and measurements
help facilitate communication, while each team
member needs specific features and capabilities
suited to their role.
TYPICAL J2EE APPLICATION
PERFORMANCE PROBLEMS
There are a wide variety of problems that can
surface during the application lifecycle. For
J2EE Web applications in production, user
experience of performance is affected by many
external network infrastructure factors that are
independent of application behavior. External
monitors can improve isolation of problems,
assisting triage. From the J2EE application
diagnostic perspective, it is essential to be
able to capture and correlate specific external
parameters, such as HTTP arguments, that can
drive performance problems in a J2EE method
or sequence of transactions. Specific latencies
and parameter captures are also frequently
needed to identify problems at J2EE interfaces
to external systems, such as backend databases,
legacy systems, and packaged software.
Within the J2EE environment, some of the most
common problems include:
1. Code problems:
a. Slow methods
• Consistently slow methods
c. Memory problems, including memory
thrashing and memory leaks
d. Coding practices, such as using
exceptions as a means to transfer
control in the application
2. Application Server configuration problems:
a. JDBC Connection pool size
b. JVM Heap size
c. Thread pool sizes
3. Architecture and design problems, with a
wide range of issues, such as:
a. Data marshalling problems resulting from
filtering at the wrong tier
b. Single-threading resulting from inadequate
synchronization design in custom code
An effective diagnostic toolset must provide
capabilities and techniques to be able to isolate
and identify the root cause of each of these
common problems, regardless of when they
emerge, from development through production.
Capturing J2EE performance data sufficient for
solving this range of problems is a significant
technical challenge.

• Intermittently slow methods, related

to specific user/data values driving
problematic application behavior

b. Synchronization problems, including

both under synchronization and over
synchronization for locks and threads

4
ALTERNATIVE TECHNOLOGIES FOR
CAPTURING J2EE PERFORMANCE DATA
There are three types of measurement interfaces
for capturing J2EE application performance data. BYTE CODE

The first is the JMX interface provided by the

JVM INSTRUMENTATION
Monitoring
SAMPLING focus:
application server vendors, exposing fixed, fairly first-level
APPLICATION       diagnostics
coarse-grained performance counters from the EVENTS

EJB Container. While useful for aggregated, AGGREGATION

APPLICATION      
high-level performance monitoring at very low SERVER
EVENTS
overhead, diagnostic capability is quite limited. JMX Deep
JVMPI TOTAL TRACE Diagnostics:
The second is a set of highly detailed, dynamic at low
overheads
data exposed by the JVM provider though JVMPI
and JVMDI interfaces. These interfaces, used
SIMPLE MONITORS PROFILERS
by code profilers, provide excellent diagnostic
Low overhead, Detailed view,
detail to developers, including specific instance fixed, coarse high overhead,
grained unsuitable
transaction tracing, but at a heavy overhead under load
price that prevents their use in load testing or
production. The third is byte-code instrumentation, Figure 1. J2EE Application Visibility: Alternatives for Capturing Performance Data

used by vendors of diagnostic tools for use in

later stages of the lifecycle.
Within byte-code instrumentation, several
techniques can be used to manage the balance
of diagnostic detail versus overhead. Sampling
a proportion of events from an event stream is
a common technique, which effectively reduces
overhead but loses the ability to reliably capture
a number of problems. If a problem occurs in an
event that isn’t sampled, or is due to a sequence
of events, sampling misses the problem.
Intermittent or sequence-related problems are
particularly intractable with this technique.
Aggregation captures and combines a sequence
of events into a single recorded data value, such
as an average. This technique reflects the impact
of all common events well, but loses granularity
for diagnosing problems related to specific data
values and individual transaction events, such as
a specific method invocation with a long latency.
Total Trace refers to a unique technology to
capture and record data for each event executed
within instrumented code. It provides the most
granular detail for diagnosing problems, such
as a method that is only slow when processing
a specific data value, a synchronization problem
when transactions contend for a resource, or
memory problems, such as thrashing or leaks.
Total Trace includes automated techniques
to narrow the scope of instrumented code, to
manage overhead.

5
DIAGNOSING PERFORMANCE PROBLEMS
WITH MERCURY FOR J2EE
Mercury for J2EE provides solutions for functional
testing, performance testing, monitoring, and
diagnosis of J2EE applications. It combines
Mercury Interactive’s industry leading application
delivery and management solutions, including
LoadRunner™ and Topaz™, with diagnostics
designed specifically for the J2EE environment.
The solutions provide a common, consistent
foundation of shared assets, metrics, scripts,
and monitors, as well as specific solutions for
the differing requirements of application delivery
and application management.
Mercury for J2EE uses a combination of JMX,
aggregation, and unique Total Trace technology,
which can capture data to diagnose all of the
types of problems described previously. It is
the only solution that can capture, under load,
necessary and sufficient information to get
DELIVERY
Performance Assurance
Platform:
LoadRunner ,
Quick Test Professional,
Test Director
Mercury Interactive Foundation
Common Diagnostics Platform:
Transaction Breakdown
Shared Scripts, J2EE Probe Technology
Figure 2: Mercury for J2EE combines J2EE-specific diagnostics with application delivery and management solutions.
to the root cause for the full range of J2EE
performance problems. Total Trace allows
capturing of every event (including arguments)
at low overheads and is distinguished from other
transaction tracing techniques, which use either
a threshold for collecting only slow events from
the event stream or a ‘tag and follow’ mechanism.
The latter techniques can show slow method
instances, but cannot show details of synchro-
nization problems or complete information for
diagnosing arbitrary memory allocation problems.
Mercury Interactive provides a combination and
choice of instrumentation techniques, to provide
both breadth of diagnostic information at low
overhead and unmatched depth, with manageable
overhead. J2EE-specific solutions in the Mercury
suite include J2EE Transaction Breakdown, Topaz™
for J2EE, and J2EE Deep Diagnostics. Each one
complements the others in providing functionality
MANAGEMENT
Mercury Interactive
Management
Monitoring, Problem
Identification:
Topaz for J2EE
Deep Diagnostics
6
best suited to the performance needs of IT users
through the lifecycle. Together, they provide the
broadest and deepest performance testing, diag-
nosis, and management capability available for
J2EE applications. Though these products are
most effective when used together throughout
the lifecycle, they are also available individually.
J2EE Transaction Breakdown for LoadRunner
The J2EE Transaction Breakdown module is
fully integrated with LoadRunner to expose and
diagnose the most common J2EE performance
problems under load testing. The same J2EE
transaction breakdown capability is included in
Topaz for J2EE, described in the following section.
J2EE Transaction Breakdown first correlates end-
user problems to the Web transaction tier, for
fast isolation and resolution, then pinpoints the
J2EE problems. J2EE Transaction Breakdown
analyzes the TCP/IP stack to break down a
transaction’s end-to-end response time to
determine whether poor transaction time is
caused by external network issues or by a specific
server tier. It then guides users through an
intuitive top-down analysis process across multiple
application layers, tracing business performance
problems from the end user all the way to the
problematic component. It shows latencies
for JSP/servlet, EJBs, classes, methods, and
JDBC connections, isolating and identifying
slow components. Aggregated data capture is
appropriate for monitoring continuously through
a load test run.
 J2EE Deep Diagnostics
J2EE Transaction Breakdown helps solve most
of the configuration and slow method problems
End-to-End Transaction
that occur in load testing and live production.
Response Time
But a significant number of the more complicated
J2EE performance problems will require deeper
diagnostics. Complex problems, such as finding

Web Page Database the cause of thread deadlocks or objects that

Breakdown Transaction
A App Server
• DNS lookup
• Time to Connect
• Time to First
buffer
• Network time
• Download time Client
• SSL handshake
• FTP
authentication
• Client time
• Error time
Figure 3. J2EE Transaction Breakdown provides response time statistics for all transactions.
Topaz for J2EE — Application Management
and Diagnostics in Production
Topaz for J2EE addresses the needs of both
operations and applications support teams by
combining real-time performance and availability
monitoring along with J2EE-specific diagnostics.
Topaz for J2EE is designed specifically for
business processes and their J2EE components
with visibility across user, transaction, application
component, and system tiers.
The solution provides the industry’s broadest
monitoring capabilities, including server,
application, and system monitoring, real-time
rapid triage functionality, and proactive alerting.
It allows 24x7 monitoring of the entire application
infrastructure, including the J2EE environment,
from a single Web-based console. Topaz for J2EE
also provides trending and correlation analysis
across all tiers, enabling continuous improvements
Web Server
Web Server App Server Database
Time Time Time
• Servlet • EJB • JDBC
• Method • Method • Connect
• JNDI lookup • Execute
• SQL Query
in application performance and availability.
Agentless monitoring for most infrastructure
components can optimize the total cost of systems
management. In addition, it can facilitate
capacity planning by ensuring that purchases
of hardware and software are optimized.
Topaz for J2EE provides correlation of end-user
performance with root cause in the infrastructure
and application layers. Its J2EE-specific
diagnostics identify problems involving EJB
components, methods, and JDBC calls with SQL
statements. J2EE diagnostics integrated with
Topaz for J2EE provides the same transaction
breakdown capabilities as LoadRunner, using
aggregated data capture, while optimized for
low-overhead, real-time monitoring, triage, and
problem isolation, along with configuration tuning
information such as heap utilization.
aren’t deallocated and removed by Java Garbage
Collection, are responsible for the majority of
the time and cost spent in diagnosing the root
cause of performance problems.
J2EE Deep Diagnostics uses Mercury Interactive’s
Total Trace data capture, which captures every
event in a selected area of the application with
low overhead, by providing carefully tuned, pre-
defined sets of byte-code instrumentation and
a GUI for custom selection and automatic
application of narrowly targeted instrumentation.
It is not used for 24x7 production monitoring
or continuous capture in load testing runs, but
is an essential complement for focused, short-
term, deep data capture when a tough problem
is encountered. In production, a Topaz alert can
trigger a deep diagnostic data capture for offline
analysis. This stops the common frustration of
persistently recurring production performance
problems that cannot be reproduced in a test
bed. The same J2EE Deep Diagnostics works
with LoadRunner when performance problems
under load need better characterization for
effective bug definition and rapid fixes.
J2EE Deep Diagnostics provides multi-layer
correlation of HTTP, servlets, JSPs, EJBs/objects,
methods, and SQL calls. It correlates these
internal component measurements with JMX
and OS metrics to show problem details. It
delivers graphical views of transaction traces, call
chains, memory leaks and thrashing problems,
synchronization details of locks held and threads

7

Figure 4. J2EE Deep Diagnostics root cause analysis for memory problems.
blocked, latency charts, and source code views
of problem methods. J2EE Deep Diagnostics
captures details of specific instances of method
invocations, including parameters passed and
individual latencies, to pinpoint the source of
intermittent problems caused by specific user
data values or usage patterns.
An example of the root cause analysis capability
is illustrated in Figure 4. This screen is showing
memory problems with the Live Object Distribution
report, selectively captured for any Java objects —
not just Java collections, as provided by another
vendor’s tool. It shows three methods that are
probably leaking and one that is thrashing.
Additional drill-down screens show the lines of
code where objects are being allocated.
For further examples and details on how
synchronization problems, intermittent slow
methods, and other J2EE performance problems
are solved using the Mercury for J2EE solution,
contact your Mercury Interactive representative.
J2EE Deep Diagnostics provides the level of infor-
mation expected from a code profiler, collected
selectively from an application under heavy load
that makes code profilers unusable. This results
in unique application visibility for diagnosing
the toughest J2EE performance problems in
load testing, staging, and production.
8
SUMMARY
With Mercury for J2EE, enterprises can improve
the readiness of their J2EE applications before
they are placed into production, and dramatically
accelerate problem identification, isolation, and
resolution. Mercury for J2EE allows companies
to reduce their costs of operations by providing
a business process view into the application
and by isolating and resolving problems before
business users and the bottom line are impacted.
The solutions can also help companies optimize
their existing infrastructure and avoid unnecessary
hardware or software expenditures.
The choice of the appropriate application delivery
and management tools is critical for guaranteeing
application quality and ensuring that service-
level agreements can be met. Mercury for J2EE
is the industry’s first and most complete
application delivery and management suite
for the J2EE ecosystem that improves the
quality, performance, and scalability of J2EE
applications across the entire application delivery
and management lifecycle.
Mercury Interactive —The Global Leader in BTO Software
Mercury Interactive is the global leader in business technology optimization (BTO). Our Optane suite of enterprise testing,
production tuning, and performance management solutions enable companies to unlock the value of their IT investments
by optimizing business and technology performance to meet business requirements. With Mercury Interactive, customers
can measure the quality of their IT-enabled business processes, maximize technology and business performance at every
stage of the application lifecycle, and manage their IT operations for continuous optimization throughout the lifecycle.
Our leading-edge BTO solutions, complemented by technologies and services from our global business partners, are
used by over 30,000 customers—including 75% of the Fortune 500—to improve quality, reduce costs, and align IT
with business goals.

Mercury Interactive, Topaz, TotalTrace, and LoadRunner are registered trademarks or trademarks of Mercury Interactive Corporation or its wholly-owned subsidiaries,
Freshwater Software, Inc. and Mercury Interactive (Israel) Ltd. in the United States and/or other countries. All other product and company names are used herein
for identification purposes only, and may be trademarks of their respective companies. © 2003 Mercury Interactive Corporation. Patents pending. All rights reserved.
WP-0???-0903

M E R C U R Y I N T E R A C T I V E C O R P O R AT E H E A D Q U A R T E R S
1325 Borregas Avenue, Sunnyvale, CA 94089 U.S.A. Phone: 408-822-5200 or 800-837-8911
www.mercuryinteractive.com