Level 2, 222 Pitt Street Level 28, 303 Collins Street

ABN: 99 070 946 986 Sydney NSW 2000 Melbourne VIC 3000
www.qualtech-int.com.au Phone: (02) 9264 0055 Phone: (03) 9614 8555
email: info@qualtech-int.com.au Fax: (02) 9264 0033 Fax: (03) 9614 8666

QualTech International

LoadRunner User’s Guide

A White Paper by Ivan Maclaine

Contents
Introduction
LoadRunner Overview
Preparation
Development
Execution
Analysis
Results

Software Quality Technicians

QualTech INTERNATIONAL PTY LTD LoadRunner User’s Guide

Introduction
Purpose
The Client/Server approach to providing business solutions, while at the same time
delivering more functional and powerful systems, is also introducing new risks. As
well as the functional concerns associated with any computer systems, client/server
solutions have a much greater associated risk in terms of the performance of the overall
system. LoadRunner is a software tool that facilitates in measuring the performance of
systems prior to deployment. This allows performance issues to be addressed before
rollout.
This paper is a guide to implementing a LoadRunner based load-testing solution. It
also provides some theoretical background into the area, defining several key concepts.
It is recommended that this paper be used in conjunction with the LoadRunner
Implementation Guide available from Mercury Interactive. This paper should not be
seen to replace Mercury’s guide, but as a supplement to it.

Load/Stress Testing
Load and stress testing are two different, but related forms of testing. Stress testing
involves testing a system under extreme cases, which will rarely occur under normal
conditions. Such testing can involve such scenarios as running many users
concurrently for short times, short transactions repeated quickly, and extraordinarily
long transactions.
Load testing places a more realistic load on the system. Scenarios are modelled on the
demand that the system is expected to face under real conditions. Load testing starts by
placing a low demand on the system, and gradually increasing the load. This process
allows measuring the performance of the system under different loads, as well as
determining the maximum load that under which the system can respond acceptably.
Load and stress testing are clearly two interrelated processes, generally performed
concurrently.
It is important to note what load/stress testing isn’t. They are not functional tests. In
order to perform performance testing, it must be assumed that a reasonably stable and
functional system is in use, otherwise results are meaningless. With that said, it is still
common to uncover functional problems in a system while performing load testing.
However, uncovering excessive functional problems can be a sign of inadequate
functional testing.
Black Box Testing
Strictly speaking, black box testing involves testing a system as a whole, not as a
system of connected parts. Realistically, however, black box testing still provides some
differentiation between clients, the server, and the network connecting them. This
means that when performing black box testing, it is possible to determine whether time
is being spent in the client, the network, or the server, and hence determine where
bottlenecks lie.
The limitation of this approach is that knowing, for example, that the server is the
bottleneck of a system is not necessarily sufficient. Is the bottleneck caused by a lack

Software Quality Technicians Commercial-in-confidence

QualTech INTERNATIONAL PTY LTD LoadRunner User’s Guide

of memory, a lack of CPU performance, poor algorithms, or one of many other causes?
To answer these questions, diagnostics need to be performed on the server while
generating a load.
White Box Testing
White box testing treats the system as a collection of many parts. During white box
testing, many diagnostics will be run on the server(s), the network, and even on clients.
This allows causes of bottlenecks to be much more ably identified, and therefore
addressed.
White box testing requires much greater technical knowledge of the system than does
black box testing. To perform white box testing, knowledge is needed as to the
components that make up the system, how they interact, the algorithms involved with
the components, and the configuration of the components. It must also be known how
to perform diagnostics against these components, and even what diagnostics are
needed and appropriate. This knowledge can generally only be provided by someone
intimate with the system.
A problem with white box testing is the effect that running diagnostics has on system
performance. Ideally, running diagnostics should have no impact on performance.
However, with the exception of the network, this is rarely possible, running diagnostics
consumes some system resources. When considering the appropriateness of various
diagnostics, one must consider the benefit vs. the impact on performance.
Black Box vs White Box Testing
Black box testing is generally much easier to perform than white box testing, requiring
system-specific expertise. However, while black box testing may identify that
problems do exist, white box shows with much more clarity where they exist.
Black box simply shows whether or not performance is acceptable. This is still useful,
as it can show which transactions are taking the most time, as well as highlight
problems with certain numbers of transactions. A common use of black box testing is
to identify knees in the response of a system against the number of concurrent
transactions. Such a knee indicates a limitation in the system that once passed,
provides exponentially decreasing system performance, a bad thing if reached.
End to End Testing
End to end testing is testing from the point of view of the client. It involves manually
or automatically using the actual client software to perform business processes, and
measures the response time of the application. End to end testing measures system
performance from the end users perspective
LoadRunner Overview
LoadRunner is a tool, or more accurately a collection of tools, which facilitates in
performing load testing on a system. LoadRunner is used to create a load or stress on a
system, and can be used to measure the responsiveness of the system to that load. The
key concept behind LoadRunner is the use of virtual users, or vusers.
Virtual Users

Software Quality Technicians Commercial-in-confidence

QualTech INTERNATIONAL PTY LTD LoadRunner User’s Guide

To perform a load test, a system needs many users to access the system concurrently,
often one hundred or more. Performing such a test with people is a logistical
nightmare. When considering that the test cycle is generally iterative, this problem is
compounded.
LoadRunner overcomes this problem through the use of vusers. A vuser performs the
actions that a real user would perform, except that a vuser is controlled by
LoadRunner, and the actions are performed automatically.
Vusers are created to perform specific transactions in the system. Once created a vuser
can be replicated as many times as required, thus producing an effective load as large
as is required. With the exception of GUI vusers, virtual users don’t use the client to
perform their actions; they take the place of the client. This allows in many vusers to
be run on one workstation, as the overhead associated with the client is not required.
This does mean, however, that non-GUI vusers do not test the performance associated
with the client side of a system, only the network and server associated performance.
Vusers can be of four main types.
DB Virtual Users
A DB, or database vuser performs operations against a database server. LoadRunner
supports all the most popular interfaces to database servers including but not limited to
ODBC, SQL, and various Oracle interfaces. It is also possible to create LoadRunner
interfaces that allow recording against unsupported databases.
Recording a DB vuser is as simple as performing a business process while having
launched the LoadRunner Virtual User Generator. This creates a script which is used
by LoadRunner to perform the recorded business process.
Once recorded, a vuser script is then parameterised. This process involves replacing
the hard coded data values in the script with parameters, which refer to tables external
to the script. Values can then be placed in these tables allowing vusers to perform a
single business process, but with different data each time.
Web Virtual Users
A web virtual user is one that performs actions against a web based system. A web
vuser is essentially identical to a DB vuser, except that where the DB vuser is recorded
against a client application that communicates directly with a database of some
description, a web vuser is recorded against a web based application.
RTE Virtual Users
Remote Terminal Emulation virtual users are used to emulate users that would be
operating on a lightweight, character based terminal.
SAP Virtual Users
SAP R/3 virtual users are created somewhat differently than the others, and require the
SAP R/3 LoadRunner add-on. Essentially, a SAP R/3 virtual user is one that performs
actions in a SAP R/3 environment.

Software Quality Technicians Commercial-in-confidence

QualTech INTERNATIONAL PTY LTD LoadRunner User’s Guide

GUI Virtual Users

Unlike the previous types of vusers, GUI virtual users actually perform actions using
the client. This means that fewer GUI vusers can be run per workstation than the other
types of vusers due to the overhead of the client, and is in fact limited to one GUI
vuser per PC workstation.
GUI vusers are used not so much to generate load on a system, but to test end-to-end
performance associated with various business processes. Although LoadRunner can
control GUI vusers, it is unable to create them. For this purpose, WinRunner is
required.
Controller
The LoadRunner controller is used to create and run scenarios. Once vuser scripts have
been created, they can be added into the controller in varying quantities to produce the
demand on the system that is required.
When running a scenario, the controller coordinates all the vusers, graphical or
otherwise. It starts vusers as required, and can force them to rendezvous with each
other, thus producing a maximum peak demand.
Agent
When running a scenario, every workstation that is to be used to generate load must
have the agent software installed. This software allows a remote (or local, but this is
not advised) controller to send vusers to the workstation to be run. The agent
communicates directly with the controller, reporting performance data of the vuser, as
well as ensuring that the vuser runs as specified in the scenario.
Virtual User Generator
The virtual user generator (VUGen) is used to generate scripts for all non-GUI vusers.
The VUGen records the behaviour of the client software, and automatically generates a
script that can be used by the controller to run vusers.
Once a script is created, the VUGen can then be used to parameterise the script. This
involves replacing hard-coded values in the script with references, which refer to
external tables. The VUGen provides a semi-automated procedure for doing this.

Load Analyser
The load analyzer utility presents the results of running a scenario in a variety of
graphical and tabular forms. These graphs and tables can be used to determine the
acceptability of a system, and to determine areas of concern. The data presented can be
exported in a variety of popular formats, allowing it to be used outside LoadRunner.
Preparation
Before load testing can begin, a certain amount of preparation is needed, however the
exact time required varies considerably from system to system. A guide to the issues
that should be addressed follows.

Software Quality Technicians Commercial-in-confidence

QualTech INTERNATIONAL PTY LTD LoadRunner User’s Guide

Planning
This phase identifies the goals of testing, how the goals may be reached, and a timeline
for doing so. A typical goal is to conduct load tests to determine where bottlenecks lie
in a system, and to suggest ways the system could be improved to overcome these.
Note here that this goal implies some degree of white box testing, as it asks us to
determine where the bottlenecks lie inside the system.
In this phase, resources are allocated for the load test. This includes people, hardware
and time. Project managers in conjunction with testing consultants generally perform
this phase. This stage can generally be performed quickly, in no more than a few days.
Analysis/Design
This phase involves identifying transactions performed in a system, rates of performing
transactions, and scenarios that model expected use of the system. It involves
performing a risk assessment of all transactions of a system to determine which are the
most critical, in terms of:
Frequency of use
Demand placed on system
Importance to the system
Any other criteria deemed appropriate
A transaction is a series of steps a real user would perform to achieve a business
process in the system. Typical transactions include such things as performing a
customer query, entering an order, and invoice printing. The exact transactions, and
complexity thereof vary considerably between systems. It is preferable to limit the
complexity of transactions, especially compound transactions, while ensuring that
meaning isn’t lost. This is for practical reasons; complex transactions are more difficult
to model, maintain, and reduce flexibility when creating scenarios.
It is important to determine exactly what transactions should be used in load testing;
creating them all as vuser scripts is time consuming and inefficient. Experience shows
that modelling 80% of a user population provides accurate load tests, and this is
typically achieved by about 20% of transactions. This typically amounts to 5-10
transactions per application module, although this figure can vary depending on the
modularity of a system.
The previously mentioned risk assessment is used to determine which transactions
should be used in the load test. When determining the transactions to implement, the
tradeoff of accuracy of the load test against the time involved in getting results needs
addressing. The risk assessment is best performed by both project managers and
experienced testing consultants, and generally takes two or more days.
Once identified, the transactions to be performed during load testing must be detailed
in a step-by-step fashion. These can later be passed on to test engineers who can then
work unassisted to create LoadRunner vusers. When detailing the transactions, it is
advised to extract the data that can change between runs of the transaction from the rest
of the script. This makes it clear to the test engineer exactly what is to be

Software Quality Technicians Commercial-in-confidence

QualTech INTERNATIONAL PTY LTD LoadRunner User’s Guide

parameterised.
Instead of detailing transactions in the above fashion, it is also possible to make
available to test engineers people knowledgeable with the system that can show them
how to perform transactions as required. This approach requires less planning, and up-
front work, however is more likely to lead to transactions which do not accurately
reflect the processes of real uses.
With transactions identified, it is also necessary to design the scenarios to be run. The
scenarios should be designed with the goals of the load testing in mind, and place
demand on the server and network as required to meet the goals of the test.
Data Dependencies and Consumption
Transactions can be divided into two categories; those that do not have any need of
information already in the system, and those that do. The first type are easily
implemented, but are in a minority. These transactions can simply be recorded,
parameterised as appropriate, and run as many times as required.
The second type has a dependency on other data in the system. This dependency can be
quite weak, such as simply needing an unused customer ID number, or as strict as
requiring a specific customer ID number. There are also transactions which consume
data. Consider a delete customer transaction; it is dependent upon a pre-existing
customer ID, but once the transaction is performed with that ID, that ID becomes
unavailable, it has been consumed.
It is important that in this analysis phase, that data dependencies and data consumption
are identified. This allows the system to be pre-populated with sufficient data that
scenarios can be run several times before requiring a refresh.
Testing Environment
The environment used for testing can vary considerably depending on the goals of the
load test. If the load test aims to simply determine whether the performance of a server
is satisfactory, then the test environment can consist of the server and a high-powered
workstation connected by a high-speed network.
If, however, the impact of the network is a larger concern, then it is necessary to have a
test environment that runs over a network identical to the one used when the system
goes live. This is especially a concern when testing a system that will be run over a
WAN.
Isolated Environment
Generally, it is preferable to have a testing environment that is isolated from other
users. This allows precise measurement of the impact of different loads on the system,
with no glitches caused by external network or server access.
However, in systems that are to be run over the same network as other systems, which
will have users accessing the server for reasons outside the system under test, this is
not always desirable. In these cases, it may be preferable to perform load testing with
the test environment connected to other systems, as it would be when live. The danger
of this approach is that spikes can enter into performance measures, and that the load

Software Quality Technicians Commercial-in-confidence

QualTech INTERNATIONAL PTY LTD LoadRunner User’s Guide

test can render other activities unusable, due to network and server congestion.
An isolated environment is recommended for testing unless there are very good
reasons to the contrary.
Realistically sized database
In many cases, the responsiveness of a system is related to the amount of data in the
system. Therefore, it is meaningless to perform load tests on a system unless it contains
a realistic amount of information. In this phase, it is important to identify the expected
size of the system when it goes live, so that it can be later populated to reach this size.
Identify LoadRunner Workstations
Depending on the exact goals of the load testing, a number of workstations to run
vusers must be identified. If the load test aims to do no more than place a heavy load
on the server, then one high powered workstation is all that is required. If the effects of
various network connections is to be tested, then workstations will be required at the
end of these connections, or least at the end of identically simulated connections.
LoadRunner Installation
At this point, LoadRunner will need to be installed on at least one machine for
development purposes. This install will include all components of LoadRunner.
Workstations that have been identified as those to run vusers can be installed with the
LoadRunner agent at this stage, but this is not necessary until just before execution of
the scenarios.
Development
Development is one of the more time consuming aspects to load testing. Development
can easily take one month or more, but this depends entirely on the number and
complexity of the transactions to be modelled, as well as the number of people
working in development.
A proficient test engineer is generally able to produce about 2 scripts per day, with less
capable and experienced people capable of about half this.
LoadRunner Scripts
LoadRunner vuser scripts form the backbone of a load test. Every transaction must be
recorded, parameterised, and tested prior to executing load scenarios.
Recording Vuser scripts
Recording vuser scripts is normally as simple as performing actions in the client
application while recording with the virtual user generator. The VUGen creates a script
file containing all the data access that occurs between the client and the server.
In some cases, this is not sufficient, as the client application may communicate with an
application such as Excel, which then communicates with the server. The VUGen does
not record this secondary communication. If this is the case, it is possible that an
experienced test engineer can first record against the client, and then against the
secondary application, and combine the two scripts. This is a complicated procedure,

Software Quality Technicians Commercial-in-confidence

QualTech INTERNATIONAL PTY LTD LoadRunner User’s Guide

and should not be attempted by inexperienced test engineers.

Parameterising Vuser scripts
Once a vuser script is recorded, it must then be parameterised. This involves extracting
the hard-coded values from within the script, and placing them in an external data
table. This allows a single vuser to be run many times, and to use different data on
each run. This provides a more realistic demand on the system, as it prevents the server
from caching the results of many transactions.
Scripts can be parameterised in a variety of ways. Some parameters will be defined to
act as global variables, specifying such things as the database name, while others will
take on a new value on every iteration of the vuser. LoadRunner is extremely flexible
in this respect, and can model most requirements that may be placed on the load test.
Testing Scripts
Test scripts, like any form of program, need to be tested. Initial testing involves
running a test in isolation, ensuring that it performs the transaction as intended, that no
unforeseen data is consumed, and that all necessary data is available for it.
Once this is achieved, multiple instances of a test are run concurrently, ensuring that
they don’t conflict with each other, and that their parameters are correctly set so as not
to cause deadlocks. It is also preferable to run different scripts concurrently, ensuring
that one script cannot cause another to deadlock. If the analysis phase was performed
with due care, this will not be a problem.

WinRunner Scripts
WinRunner scripts are not used so much to generate load, but to measure the end to
end performance of the system when it is under load. As a result of this, it is common
for only a subset of LoadRunner implemented transactions to be implemented as
WinRunner scripts. This is a practical consideration; scripting all transactions as both
GUI and non-GUI vusers is time consuming. However, it is important that sufficient
transactions are scripted using WinRunner so as to provide a clear assessment of the
end to end performance of a system, and occasionally this does involve scripting all
transactions. The tradeoff here is again the time required vs. benefits gained in terms of
the usefulness of the performance results.
Scripting transactions as GUI vusers is considerably different to non-GUI vusers, and
is explained fully in the associated WinRunner implementation documentation.

Data Preparation
Preparation of data in the system must be completed before scenario execution can
commence. This phase ensures that the system contains enough data that the most
demanding load scenarios can be executed at least once, but preferably three or more
times, before the system requires refreshing.
This phase can be performed in parallel to script development, and can normally be
achieved quite quickly if the data demands of the transactions was correctly identified
in the planning phase.

Software Quality Technicians Commercial-in-confidence

QualTech INTERNATIONAL PTY LTD LoadRunner User’s Guide

Scenario Creation
With transactions scripted as both GUI and non-GUI vusers, scenarios that place the
system under load can be implemented. These scenarios should have been previously
identified in the earlier planning phase, and all that is required here is creating them in
the LoadRunner controller.
As a rule of thumb, most scenarios should run for about one hour. This allows the
results of the run to be more statistically reliable than a shorter run, and is not so long
as to consume too much time. Shorter scenarios should be looked upon unfavourably,
and longer scenarios implemented only if there is good cause to do so.
Golden Area
Before execution is commenced, it is important to create a golden area, or a place
where all scripts, scenarios, and a database baseline can be placed. This golden area
should be treated as write-once read-many. This means that scripts placed here must be
bug-free, as modifications made to files in the golden area can greatly confuse future
testing.
The database baseline should contain all the data required for performing the most
strenuous of the load tests, and is used to refresh the system as required. Preferably, the
database baseline placed here should allow several load tests to be performed before
requiring the system to be refreshed. As these refreshes are generally time consuming
activities, it is ideal if refreshes can be performed overnight, after every day of load
testing.
It is important to note that it is not always possible or feasible to create database
baselines, due to the complexity of a system and its batch processing. Where this is the
case, it is important to pre-determine this, and make allowances in creating the data
that load tests can consume. This type of scenario is far from ideal, as it is difficult to
compare results between runs, as too many system variables have changed. All that can
be done is to raise the issue, and continue as best as possible. It is, however, worth
noting that systems in which it is "impossible" to revert to baselines, often through
ingenuity techniques can be created allowing just that to be done.
Execution
When this phase is commenced, the testing environment must be correctly set-up, and
the scenarios for testing must be created and prioritised. The exact scenarios to be run
vary widely depending on the goals set prior to planning the load test. Some common
scenarios that are recommended follow.

Baseline Load
The baseline or light load scenario involves running small numbers of users, ensuring
that the testing environment behaves as expected. As the scripts have been tested under
the development environment, problems here are likely to indicate configuration
problems in the testing environment. If these occur, they must be addressed before
more demanding testing can begin.
A second reason for executing a light load scenario is to use the results as a
benchmark. Performance results obtained under the baseline load should be more than

Software Quality Technicians Commercial-in-confidence

QualTech INTERNATIONAL PTY LTD LoadRunner User’s Guide

satisfactory, otherwise the system has no hope of performing adequately under a higher
load.
Assuming no performance problems were noticed during this scenario, the results
obtained can be used as "best case" results. These results indicate how the system
performs when it is not under any real form of stress, allowing conclusions to be drawn
about the performance of the system during the higher load tests.
Medium Load
Depending exactly on the requirements of the load testing project, a medium load may
constitute what is considered a real-world worse case load, or as little as 50% of this.
Where the load that is applied is less than the expected worse case load, the scenario is
used to identify major failings in a system, but does so in a way that doesn't highlight
many of the more minor failings, allowing an easier analysis of results. Where the load
is equivalent to the expected real-world worse case load, actual performance of the
system can be measured, and associated problems clearly identified.
Running a medium load before moving directly to a high load allows the more major
problems of a system to be identified separately of the smaller issues, by providing a
simpler result analysis.
Heavy Load
Heavy load scenarios test a system at and beyond the worse expected demand it is
likely to face. The majority of major deficiencies in the system will have already been
identified in the medium load execution, so this phase deals more with assessing the
impact of these issues under a heavy load. However, heavy scenarios also identify
many other system bottlenecks not previously noticed, which may in fact be partially
responsible for earlier identified problems.
Heavy load scenarios are generally designed to be far more than a system can handle.
They are used not to identify if a system fails, but where it fails first, by how badly,
and why. By answering the why question, it can be determined whether a system is as
good as it should be.
Heavy scenarios often involve starting a scenario running, and incrementally adding
more and more load onto the system, until it either fails altogether, or becomes totally
unusable. At this point, the load can be slowly decremented. This allows the effect of
different loads to be easily analysed, and plotted graphically.
The analysis performed in this phase can vary depending on the exact goals and
objectives of the load testing. Results obtained from within LoadRunner are often used
in conjunction with results obtained directly from the system, where white-box testing
has been employed.
The majority of scenario execution time will be spent with heavy load scenarios, as
these provide the most information about the performance of a system.
Analysis
The entire reason of performing load testing is so a better understanding of a system’s
performance can be obtained. The results obtained during LoadRunner scenario

Software Quality Technicians Commercial-in-confidence

QualTech INTERNATIONAL PTY LTD LoadRunner User’s Guide

execution are plentiful, and this is compounded if white-box metrics have also been
measured. The analysis phase typically occurs in parallel with the execution phase, as
both phases are iterative in nature, although analysis occurs with an offset from the
execution, and can continue after execution has completed.

White Box vs Black Box Analysis

Where white-box metrics have been recorded, they can provide much more insight into
where problems exist in a system, as well as to highlight where improvements are most
necessary.
Black box analysis can show the effects of running different numbers of simultaneous
transactions, as well as showing the effect of different combinations of transactions,
but the results. However, black box testing will only highlight performance issues on a
per transaction basis.
White box analysis can show exactly where in a system time is being spent, as well as
show other information not directly related to time, such as the percentage of
transactions that result in more than 10 rows being fetched. Clearly, this information
requires a lower level of knowledge about a system than LoadRunner can gain.
Black box analysis provides the big picture of a system’s performance, which is often
all that is required. White box analysis concentrates on looking at the system as a
collection of co-operating parts, allowing finer detail to be obtained.

Results
Many load test projects iterate between the execution and analysis phases many times,
with system modifications being made between iterations, thus the load tests are used
to improve the system. Other load test projects are simply performed to measure the
performance of a system, preparing the systems owners for its performance, and
allowing them to determine if changes may be required. Depending on the situation,
the writing of results may vary.
The results of a load test project should directly relate to the original goals and
objectives of the load test. They should focus on what system requirements are met,
how well they are met, and which fail. Any issues encountered that do not directly
relate to the objectives of the project should be mentioned in full, allowing further
action to be taken where this is deemed necessary.
A write-up of results should include at least part of the analysis, but this is more often
desirable as an appendix. Load testing is often performed as part of user acceptance
testing, even if only implicitly, and this should be recognised when load test results are
being tabulated.

Software Quality Technicians Commercial-in-confidence