© 2001 Giga Information Group

Copyright and Material Usage Guidelines

May 1, 2001

Web Monitoring Solutions: Which Is the Fairest One of

All?
Jean-Pierre Garbani

Giga Position
Early on, e-business strongly emphasized availability and page download speed. Three years ago, a number
of products and services started to appear on the market, mostly centered around one “active” agent
technology — capture and playback. Since then, a number of methodologies have appeared and others are
still emerging today. Users are caught in a maelstrom of announcements and claims. Matching products to
objectives has become quite an arduous task. There are at least two fundamental reasons to capture
availability and response time in e-business. One reason is pure marketing, since client loyalty is allegedly
driven by site performance. The second is pure IT, since these metrics are a basis for infrastructure
performance management.

Giga believes that it is difficult today for a single technology to be everything to everyone since some
technologies are better suited to specific objectives. The active agent technology does not describe an actual
transaction and should, therefore, be disqualified from any projects aimed at understanding the end-user
experience. On the other hand, technologies capturing the true end-user experience may provide metrics that
are difficult to use in statistical sampling without help from specialists and may be ill suited for infrastructure
performance management.

Proof/Notes
The e-business focus on availability of site, pages, objects within a page and page download speed is rooted
in the belief that client loyalty is entirely based on performance. Of course, this is an oversimplification of the
e-commerce equation, but it nevertheless has had enough of an impact to convince a number of vendors to
offer products or services aimed at monitoring Web sites. While it seems that this was originally mostly a US
phenomenon, the growth of the European e-commerce market has resulted in a European demand for such
monitoring products. Giga estimates that the market for Web monitoring was about $300 million in 2000 —
it should grow by 48 percent in 2001 to $450 million. This estimate is limited to monitoring products and
services and excludes any performance management product of applications, servers and databases used in
Web infrastructures.

Multiple technologies (agents) are used by the different vendors. This diversity results from the perception
that IT does not have any control on the e-business client workstation. Contrary to what happens in in-house
applications, where an agent can be installed on any PC, the lack of control over the potential e-business
clients appeared as the major hurdle to overcome in monitoring Web performance. Thus, the main objective
of all subsequent technologies has been to design a clever (or not so clever) way around this problem. In
doing so, however, the very technology used may also bias the results and restrict the relevance of the data to
specific business or IT applications.

Corporate Objectives
There are two distinct objectives for e-business application monitoring, and they require distinct variations of
what seems to be the same data:

•= Business objective: The idea is to either keep the clients that have been attracted to the site and

Planning Assumption ♦ Web Monitoring Solutions: Which Is the Fairest One of All?
RPA-052001-00001
© 2001 Giga Information Group
All rights reserved. Reproduction or redistribution in any form without the prior permission of Giga Information Group is expressly prohibited. This information
is provided on an “as is” basis and without express or implied warranties. Although this information is believed to be accurate at the time of publication, Giga
Information Group cannot and does not warrant the accuracy, completeness or suitability of this information or that the information is correct.
 Web Monitoring Solutions: Which Is the Fairest One of All? ♦ Jean-Pierre Garbani

make them return on the basis of a successful experience or understand why they leave in order to
correct site performance, usability or navigation. For this type of requirement, it is important to
understand not only the download performance of each page, but also the complete client session.
The client profile and behavior when using the site are data that is useful to the business side of the
house. In effect, this is marketing root cause analysis and it goes far beyond the simple measure of
response time and availability.
•= IT management objective: Here we are talking performance and capacity management. Assuming
that the business articulated a service-level objective, the work of IT is to make sure that the site is
available and performing within limits, and if it is not, to determine why and correct the problem
(performance management), then to look at long-term issues to make sure that the site will perform
within the limits in the future (capacity management) — all that, of course, within budget. The
requirements are to have an early alert indicator of performance problems (availability and
response time) and to have the option to quickly point out the bottleneck for further analysis of
cause. Although user session and behavior may be used in modeling (capacity planning), most IT
shops at this stage are looking at being alerted and reacting in real time.

The key question is how well these requirements may be served by the technologies available.

Different Agents for Different Results

The first type of agent, and the most widely used, is the active agent. It is so termed because it actually enters
a synthetic session with the target site, as if it were an active user. Active agents use a fixed site polling
frequency (for example every five minutes) and a predetermined script. In most agents, the scripting facility
uses a record and playback method, where user navigation is recorded and becomes the basic session to be
played back and monitored. For each page requested, the agent is capable of providing a download time and a
validity check on each object of the page. Response time, threshold-based alerts can be generated and
statistics created. The simple active agent has a number of virtues and a number of drawbacks. It is very
simple to deploy and use and provides a very consistent statistical sample. From a given location, and at fixed
intervals, it will always request the same pages.

But this is precisely where it also has problems. The determination of which pages to include in the
monitoring is usually based on the perceived criticality of these pages in the overall e-business scheme, not
on user behavior. From an IT standpoint of monitoring critical resources, this is good. From a business
standpoint, it does not mean much, since it does not represent a true user experience with the site. Further, the
monitoring interval is long and allows for the monitor to either miss or amplify the long response time
associated with a traffic burst. The total interval between two measures of a given page is the polling
frequency plus the length of the session. This is ample time to either measure between bursts (and give a nice
feeling of good performance) or to monitor only bursts (and show dismal results), neither one being true.
If the monitoring stations are multiplied, to make sure that this characteristic is overcome, all these additional
sessions are going to amount to a good chunk of traffic and that will bias the results.

On the long run, however, the ups and downs will even out and the statistical sample should be
representative. The two major advantages of this type of product are the capability to alert and the availability
of good quality statistical samples for correlation with other type of data. This is typically the basis of
products such as Mercury Interactive Topaz and Empirix or services such as Keynote.

The second type of agent is the passive agent. As the name indicates, it does not create user sessions, but
records all user sessions to derive statistics on response time and user navigation. The main advantages of
such a solution are obvious: no added workload. The agent monitors what is actually going through the
infrastructure and gets a true representation of the user experience: actual response times, actual pages
requested and actual “think time” between pages. So, from this data, administrators can not only understand
response times as seen by the client, they can also recreate the client session and the users’ navigation on the
site, which is what the business is looking for. On the surface, this seems to be the ideal business-oriented

Planning Assumption ♦ RPA-052001-00001 ♦ www.gigaWeb.com

© 2001 Giga Information Group
Page 2 of 6
 Web Monitoring Solutions: Which Is the Fairest One of All? ♦ Jean-Pierre Garbani

monitoring. There are several variations on the theme, including the following:

•= Downloading a temporary agent in the client browser in order to capture all the data concerning the
user’s specific session with the site. Candle eBA (product or service) downloads a Java applet with
certain pages. The Java applet will collect data, response time and user behavior and send it to a
server where it can be stored in a database for further use. Hewlett-Packard (HP) Web Transaction
Observer uses a plug-in that is downloaded to the user browser. These solutions guarantee a real
picture of the client experience and meet all requirements for user privacy protection.
•= The second variation consists in placing an agent on the wire, at the server level for example, and
rebuilding user sessions through an analysis of the traffic and protocols. This offers advantages such
as being able to, in some products (Compuware, Mercury Interactive), understand fractional
response times, that is to distinguish between time spent on the network and in the servers. Part of
HP VantagePoint Internet Services, Tivoli Web Management and Mercury Interactive Prism are
based on this type of technique.
•= The third variation, similar to the previous one, is to package the agent into an appliance. This offers
a greater simplicity of deployment and also more scalability; Adlex UserVisibility is an example.
•= The fourth variation is a permanent agent. While everyone has been designing technologies avoiding
the permanent instrumentation of the client machine, webHancer has based its service on a
permanent agent distributed by ISPs, webHancer Companion, able to record a number of parameters
from all activities of the client. This proves to be a good source of business intelligence on client
behavior.

Although understanding the client behavior is directly beneficial to the business, the task of sifting through so
much data may prove to be too much for many IT shops. When it comes to creating a consistent statistical
sample that can be correlated with other data, the diversity of experiences, connection technologies,
geographic locations and client behavior make sorting and processing the data a tedious task. Candle,
conscious of the problem, offers eBA both as a product and as a service.

Vendor Overview
Table 1 shows an overview of some of the Web monitoring products proposed by vendors (click here to see
Table 1).

Table 2 is an overview of the Web monitoring services proposed by vendors (click here to see Table 2).

In looking at the different products and services proposed, a clear dichotomy appears:

•= Most of the products are advertised as Web management products, putting more emphasis on the
importance of response time monitoring from an IT standpoint. However, the data extracted from
passive agents can be easily turned into business-oriented data.
•= Most of the services are advertised as business oriented but are using a technology, with the
exception of Candle and webHancer, which has nothing to do with user behavior or experience.

Actually, this explains the success of log analysis products, which are products aimed at rebuilding client
sessions from the server logs. While this is possible, but imperfect, the dimension of performance is absent,
and the business side is far better served by products such as webHancer eBusiness View.

From an IT standpoint, the best products are the ones providing both passive and active agents, with the
added value of correlation with other data coming from the components. This provides good statistical
samples for capacity planning, the possibility to examine user sessions and to have fractional response times
and data correlation for rapid bottleneck identification. From a business standpoint, a passive agent is the best

Planning Assumption ♦ RPA-052001-00001 ♦ www.gigaWeb.com

© 2001 Giga Information Group
Page 3 of 6
 Web Monitoring Solutions: Which Is the Fairest One of All? ♦ Jean-Pierre Garbani

way to understand user behavior, while active agents have a limited use as health indicators. Again, services
providing competitive analysis, such as webHancer, are the most appealing solution.

Alternative View
Fundamentally, response time is response time and there are no real differences between business data and IT
data. The key is to understand whether performance is a factor in the success of the site. The fact remains that
response time is an indicator and it should be used as such by both sides of the house. Behind it, starts the real
work of understanding either why there is a performance problem (IT side) or what are the consequences
(business side). Some tools are better suited than others in providing a deeper insight into this type of
analysis, but these are secondary selection factors. The main selection choice should be data accuracy and
precision.

Findings & Recommendations

Accuracy and precision are at the core of this debate between active and passive agent technology.
IT performance and capacity management require accuracy of the statistical samples used in correlating data,
and this is achieved by active agents in a more simple fashion than with passive agents where data needs to
be sorted according to geography, user connection, etc., and will, therefore, require a strong reporting
package or service. However, when it comes to understanding why certain clients are leaving a site, passive
agents provide a superior level of precision, since they record and analyze all transactions. Giga’s
recommendation to clients include the following:

•= Understand the requirements, whether they are business oriented, IT oriented or both.
•= Understand the type of technology used by the vendor and decide if this is the best way to provide
the data required. More than one technology may be needed.
•= In the case of Web monitoring services, make sure that the vendor’s infrastructure matches the level
of accuracy required. In the case of active agents, the location of probes and their connectivity may,
in fact, bias the measures and make them both inaccurate and imprecise.
•= When selecting a product for IT, look not only at the capability to provide alerts, but also whether
the product can provide correlation and fractional response times in order to support a faster problem
resolution.

Many products promise the moon, but have difficulties reaching the tropopause. It is clear, for example, that
many Web monitoring services using active agents have been left behind in terms of technology by the more
recent passive agents when it comes to providing business-oriented data, yet this is the main target of these
services. Using them in conjunction with a log analyzer seems to bring another level of precision, but still far
from the wealth of information that could be provided by passive agents.

References
Related Giga Research
Planning Assumption, Market Overview: Infrastructure Performance Management, Jean-Pierre Garbani
Planning Assumption, Examining and Segmenting the managed Service Provider Landscape, Joel Yaffe and
Stan Schatt
Planning Assumption, Root Cause Analysis: The Next Frontier in Infrastructure Performance Management,
Jean-Pierre Garbani
IdeaByte, The Eight Second Rule: A Myth for Market Leaders, Adria Ferguson
IdeaByte, webHancer Offers E-business Intelligence, Jean-Pierre Garbani

Planning Assumption ♦ RPA-052001-00001 ♦ www.gigaWeb.com

© 2001 Giga Information Group
Page 4 of 6
 Web Monitoring Solutions: Which Is the Fairest One of All? ♦ Jean-Pierre Garbani

IdeaByte, Is Mercury Interactive Recession Proof? Uttam Narsu

Relevant Links and Other Sources

Developing Intelligent Agents for Distributed Systems: Exploring Architecture, Technologies and
Applications, Michael Knapik and Jay Johnson, McGraw-Hill, 1998

Table 1: Sample of Web Monitoring Products

Vendor Product Agent Type Technology Correlation Suitable for:

Name With Other
Data
Adlex UserVisibility Passive Protocol No IT and
Analyzer Business
BMC Patrol Active Script PB Yes IT and
Software Passive Java Applet Business
Candle eBA Passive Java Applet No IT and
Business
Computer MasterIT Active Script Yes IT
Associates Unicenter Playback
TNG
Compuware Ecosystems Active Script Yes IT
Playback
Empirix eTester Active Script No IT
(RSW) Playback
Freshwater SiteScope Active Script Yes IT
Monitors Playback
HP Openview Internet Active Script PB Yes IT
VantagePoint Services Passive Server Agent
HP Openview Web Passive Browser Yes IT and
VantagePoint Transaction Plug-in Business
Observer
Holistix Web Active Script Yes IT
Manager Playback
Mercury Topaz Active Script PB Yes IT and
Interactive Prism Passive Traffic Business
Analyzer
Tivoli STI Active Script PB Yes IT
QoS Monitor Passive Multiple
Agents
Trio Hypertrack Passive Protocol No IT and
Networks Analyzer Business
Visual Visual Active Script No IT
Networks eWatcher Playback
WildPackets EtherPeek & Passive Protocol No IT and
Webstats Analyzer Business
Source: Giga Information Group

Planning Assumption ♦ RPA-052001-00001 ♦ www.gigaWeb.com

© 2001 Giga Information Group
Page 5 of 6
 Web Monitoring Solutions: Which Is the Fairest One of All? ♦ Jean-Pierre Garbani

Table 2: Sample of Web Monitoring Services

Vendor Service Agent Type Technology Correlation Suitable for:
Name With other
data
@Watch Web Active Script No IT
Enterprise Playback
Boca AlertSite Active Script No IT
Technologies Playback
Candle eBA Passive Java Applet No IT and
Business
Compuware ecoSystems Active Script Yes IT
Playback
Envive Sensory Active Script No IT
Playback
Freshwater SiteSeer Active Script No IT
Playback
Keynote RedAlert Active URL Polling No IT
Perspective Script PB
Mercury ActiveWatch Active Script No IT
Interactive Playback
ProactiveNet Web Monitor Active URL polling No IT
Script PB
Sentire Opsis Active Script No IT
Playback
webHancer eBusiness Passive Permanent Yes Business
View Agent
Source: Giga Information Group

Planning Assumption ♦ RPA-052001-00001 ♦ www.gigaWeb.com

© 2001 Giga Information Group
Page 6 of 6