.

Agile Testing
Monday, February 28, 2005

Courtesy: agiletesting.blogspot.com

Performance vs. load vs. stress testing

Here's a good interview question for a tester: how do you define

performance/load/stress testing? Many times people use these terms
interchangeably, but they have in fact quite different meanings. This
post is a quick review of these concepts, based on my own experience,
but also using definitions from testing literature -- in particular:
"Testing computer software" by Kaner et al, "Software testing
techniques" by Loveland et al, and "Testing applications on the Web"
by Nguyen et al.

Update July 7th, 2005

From the referrer logs I see that this post comes up fairly often in
Google searches. I'm updating it with a link to a later post I wrote
called 'More on performance vs. load testing'.

Performance testing

The goal of performance testing is not to find bugs, but to eliminate

bottlenecks and establish a baseline for future regression testing. To
conduct performance testing is to engage in a carefully controlled
process of measurement and analysis. Ideally, the software under test
is already stable enough so that this process can proceed smoothly.

A clearly defined set of expectations is essential for meaningful

performance testing. If you don't know where you want to go in terms
of the performance of the system, then it matters little which direction
you take (remember Alice and the Cheshire Cat?). For example, for a
Web application, you need to know at least two things:

• expected load in terms of concurrent users or HTTP connections

• acceptable response time

Once you know where you want to be, you can start on your way there
by constantly increasing the load on the system while looking for
bottlenecks. To take again the example of a Web application, these
bottlenecks can exist at multiple levels, and to pinpoint them you can
use a variety of tools:

• at the application level, developers can use profilers to spot

inefficiencies in their code (for example poor search algorithms)
• at the database level, developers and DBAs can use database-
specific profilers and query optimizers
• at the operating system level, system engineers can use utilities
such as top, vmstat, iostat (on Unix-type systems) and PerfMon
(on Windows) to monitor hardware resources such as CPU,
memory, swap, disk I/O; specialized kernel monitoring software
can also be used
• at the network level, network engineers can use packet sniffers
such as tcpdump, network protocol analyzers such as ethereal, and
various utilities such as netstat, MRTG, ntop, mii-tool

From a testing point of view, the activities described above all take a
white-box approach, where the system is inspected and monitored
"from the inside out" and from a variety of angles. Measurements are
taken and analyzed, and as a result, tuning is done.

However, testers also take a black-box approach in running the load

tests against the system under test. For a Web application, testers will
use tools that simulate concurrent users/HTTP connections and
measure response times. Some lightweight open source tools I've used
in the past for this purpose are ab, siege, httperf. A more heavyweight
tool I haven't used yet is OpenSTA. I also haven't used The Grinder
yet, but it is high on my TODO list.

When the results of the load test indicate that performance of the
system does not meet its expected goals, it is time for tuning, starting
with the application and the database. You want to make sure your
code runs as efficiently as possible and your database is optimized on
a given OS/hardware configurations. TDD practitioners will find very
useful in this context a framework such as Mike Clark's jUnitPerf, which
enhances existing unit test code with load test and timed test
functionality. Once a particular function or method has been profiled
and tuned, developers can then wrap its unit tests in jUnitPerf and
ensure that it meets performance requirements of load and timing.
Mike Clark calls this "continuous performance testing". I should also
mention that I've done an initial port of jUnitPerf to Python -- I called
it pyUnitPerf.

If, after tuning the application and the database, the system still
doesn't meet its expected goals in terms of performance, a wide array
of tuning procedures is available at the all the levels discussed before.
Here are some examples of things you can do to enhance the
performance of a Web application outside of the application code per
se:

• Use Web cache mechanisms, such as the one provided by Squid

• Publish highly-requested Web pages statically, so that they don't
hit the database
• Scale the Web server farm horizontally via load balancing
• Scale the database servers horizontally and split them into
read/write servers and read-only servers, then load balance the
read-only servers
• Scale the Web and database servers vertically, by adding more
hardware resources (CPU, RAM, disks)
• Increase the available network bandwidth
Performance tuning can sometimes be more art than science, due to
the sheer complexity of the systems involved in a modern Web
application. Care must be taken to modify one variable at a time and
redo the measurements, otherwise multiple changes can have subtle
interactions that are hard to qualify and repeat.

In a standard test environment such as a test lab, it will not always be

possible to replicate the production server configuration. In such
cases, a staging environment is used which is a subset of the
production environment. The expected performance of the system
needs to be scaled down accordingly.

The cycle "run load test->measure performance->tune system" is

repeated until the system under test achieves the expected levels of
performance. At this point, testers have a baseline for how the system
behaves under normal conditions. This baseline can then be used in
regression tests to gauge how well a new version of the software
performs.

Another common goal of performance testing is to establish

benchmark numbers for the system under test. There are many
industry-standard benchmarks such as the ones published by TPC, and
many hardware/software vendors will fine-tune their systems in such
ways as to obtain a high ranking in the TCP top-tens. It is common
knowledge that one needs to be wary of any performance claims that
do not include a detailed specification of all the hardware and software
configurations that were used in that particular test.

Load testing

We have already seen load testing as part of the process of

performance testing and tuning. In that context, it meant constantly
increasing the load on the system via automated tools. For a Web
application, the load is defined in terms of concurrent users or HTTP
connections.
In the testing literature, the term "load testing" is usually defined as
the process of exercising the system under test by feeding it the
largest tasks it can operate with. Load testing is sometimes called
volume testing, or longevity/endurance testing.

Examples of volume testing:

• testing a word processor by editing a very large document

• testing a printer by sending it a very large job
• testing a mail server with thousands of users mailboxes
• a specific case of volume testing is zero-volume testing, where
the system is fed empty tasks

Examples of longevity/endurance testing:

• testing a client-server application by running the client in a loop

against the server over an extended period of time

Goals of load testing:

• expose bugs that do not surface in cursory testing, such as

memory management bugs, memory leaks, buffer overflows, etc.
• ensure that the application meets the performance baseline
established during performance testing. This is done by running
regression tests against the application at a specified maximum
load.

Although performance testing and load testing can seem similar, their
goals are different. On one hand, performance testing uses load
testing techniques and tools for measurement and benchmarking
purposes and uses various load levels. On the other hand, load testing
operates at a predefined load level, usually the highest load that the
system can accept while still functioning properly. Note that load
testing does not aim to break the system by overwhelming it, but
instead tries to keep the system constantly humming like a well-oiled
machine.
In the context of load testing, I want to emphasize the extreme
importance of having large datasets available for testing. In my
experience, many important bugs simply do not surface unless you
deal with very large entities such thousands of users in repositories
such as LDAP/NIS/Active Directory, thousands of mail server
mailboxes, multi-gigabyte tables in databases, deep file/directory
hierarchies on file systems, etc. Testers obviously need automated
tools to generate these large data sets, but fortunately any good
scripting language worth its salt will do the job.

Stress testing

Stress testing tries to break the system under test by overwhelming its
resources or by taking resources away from it (in which case it is
sometimes called negative testing). The main purpose behind this
madness is to make sure that the system fails and recovers gracefully
-- this quality is known as recoverability.

Where performance testing demands a controlled environment and

repeatable measurements, stress testing joyfully induces chaos and
unpredictability. To take again the example of a Web application, here
are some ways in which stress can be applied to the system:

• double the baseline number for concurrent users/HTTP connections

• randomly shut down and restart ports on the network
switches/routers that connect the servers (via SNMP commands for
example)
• take the database offline, then restart it
• rebuild a RAID array while the system is running
• run processes that consume resources (CPU, memory, disk,
network) on the Web and database servers

I'm sure devious testers can enhance this list with their favorite ways
of breaking systems. However, stress testing does not break the
 system purely for the pleasure of breaking it, but instead it allows
testers to observe how the system reacts to failure. Does it save its
state or does it crash suddenly? Does it just hang and freeze or does it
fail gracefully? On restart, is it able to recover from the last good
state? Does it print out meaningful error messages to the user, or does
it merely display incomprehensible hex codes? Is the security of the
system compromised because of unexpected failures? And the list goes
on.

Conclusion

I am aware that I only scratched the surface in terms of issues, tools

and techniques that deserve to be mentioned in the context of
performance, load and stress testing. I personally find the topic of
performance testing and tuning particularly rich and interesting, and I
intend to post more articles on this subject in the future.
posted by Grig Gheorghiu at 7:33 AM

73 Comments:

• Perhaps of interest:

http://openload.sourceforge.net/

By Anonymous, at 2:16 PM

• Thank you for this nice overview. I would have liked JMeter to be mentioned along
with OpenSTA and TheGrinder.

By falkenbt, at 7:15 AM

• Thnaks for sharing very information on all three aspects of testing: Performance,
load and stress testing.

By Anonymous, at 8:36 AM
• You can look at Load Testing Terminology by Scott Stirling

Still don't see a clear cut difference between performance and load testing, they look
more like synonyms for me with some small difference in, perhaps, goals of testing.

By Alex, at 5:53 PM

• Hi,

I still didnt get clear abt the difference btn performance and load testing. What are
the parameters we should consider in both testings? Can we run single user/single
iteration for performance? Is it neccessary to increase the load? Can we run the
performance scripts for a fixed time?

Clearing this willbe very grateful.

Regds
Senthil

By Senthil, at 4:55 AM

• Senthil,

I usually start my performance testing with a single client machine and I constantly
increase the number of simulated concurrent users -- for example, I start with 10
concurrent connections to the server and I increase that number to 100 in
increments of 10. At some point, either the client or the server will become the
bottleneck. If the server is capable of serving a large number of concurrent users,
then the client machine will become the bottleneck. At that point, you need to add
more clients to your test and continue to increase the load on the server.

As for the main difference between performance and load testing: you do
performance testing in order to find any bottlenecks in your application code and
eliminate them. When you can't possibly optimize your application anymore, you
start doing load testing, which tells you for example how many Web servers you
 need behind a load balancer, how many database servers you need, etc. -- all this so
that you can sustain some pre-defined load that your customers/users require.

By Grig Gheorghiu, at 6:56 AM

• Hi Grig Gheorghiu,

Thanx for your explanation.But I have one more question to you.

Let me explain the way how I am doing the performance/load(I dont know how to
name it) testing.
We are doing performance testing of a web application from a client machine.
steps:
1) first I will prepare the test case scenarios for the testing based on the need of
customer(for example some searches, Reports and some navigations)
2) I will record the steps using OpenSTA.
3)Develop the test task and run the individual tests from 1 v.user to max of 100
users.
4)Before reaching 100 v.users only I will get errors in the results and graph start
decreasing,
5)So based on this graph and data I will find out the bottleneck for that scenario.
6) usually we are running each test for a constant period like 10mts or 15 minutes.
7)Here how I can distinguish btn client and server bottleneck.
8) Am I doing the testing in a right way or not. We are calling this as performance
testing only.
9) Here how to do load test?

Sorry for putting lot of queries.

Thanks and regds,

Senthil

By senthil, at 11:24 PM

• Senthil,
 As I said before, it's sometimes hard to distinguish between performance and load
testing. Many people use them interchangeably and I can't really blame them.

One of the points I was trying to make in my blog entry was that by performance
testing people generally mean tuning your application so that you eliminate
bottlenecks in your code. In your example, you would need to profile your Web app
while you are sending requests from your client. You will then be able to see what
functions/methods in your code are the hot spots (i.e. where most of the time is
spent). At that point, you can try optimizing your code, or maybe you database
access, if the code involves calls to the database.

If there's no possible way to further optimize your code/database logic, then you
need to look at scaling the application either vertically (by throwing more hardware
at the server), or horizontally (for ex. by adding more Web servers behind a load
balancer).

Load testing is what you're doing after your eliminated the bottlenecks in your code.
You want to know at that point what the maximum capability of your application is.
So I think in your case, since it doesn't seem like you're doing any
profiling/benchmarking within the application, we can say you're doing load testing.
Your methodology is sound, but at the same time you need to monitor the server via
OS-specific utilities (vmstat, top, iostat or Windows Perfmon) and also, as I said, you
need to monitor your app via a profiler.

As to how to see whether the client or the server is the bottleneck, you can try
running another network-bound process on the client against a different server. If the
client can sustain the second process, it means the first server was the bottleneck. If
the resources on the client are exhausted, it means the client is the bottleneck. For
more info on client-related bottlenecks, see
http://www.hpl.hp.com/personal/David_Mosberger/httperf/doc004.html

By Grig Gheorghiu, at 10:24 AM

• Hi Grig Gheorghiu,
 Thanks for your clean explanation.
Actually Our work starts from discussing with the technical team for getting the right
scenarios where we need to do performance testing. After getting the results we will
do the analysis using the HTTP DATA List in Excel sheet.
For CPU monitoring we are using a tool called "PerfView" where we can get all server
(CPU time,network,I/O usuage and etc)related parameter values. We will get both
app server and db server details in that tool. My Question is running for a constant
period is right or not. I feel this time is need to scaleup the vusers in server. But I am
not sure how we can decide like running for 10 mts or 15mts. Will it make any
change in results. We are taking the average time of each urls in excel sheet started
from 1 user to max users.

Clearing the time duration to run will make me comfort. I am also looking your other
guidance points which definitly will improve our quality and use of testing.

Thanks and Regds,

Senthil

By senthil, at 11:55 PM

• Good concept on Load and Performance testing.

Now can we say load testing is some thing where we test the systems to check how
they work under specified load?
and performance testing is testing an application with slow ramp up to see the
response times under the load?

Correct me if i am wrong. The terms

Performance,Volume,Load,Spike,
stress testing always confuse me.

Suresh Chatakondu

By Anonymous, at 1:49 PM
• Suresh, I think you're right in your characterization. I'd also add that for
performance testing, you need to be profiling your app and monitoring your servers
while you're increasing the load on the system. I tried to explain these terms in a
better way in another blog entry: http://agiletesting.blogspot.com/2005/04/more-
on-performance-vs-load-testing.html

By Grig Gheorghiu, at 1:52 PM

• rssd

By rajkumar, at 1:53 AM

• Hi

I have still confussion with Performance,Load Testing, Volume Testing Can you plz
one example for each testing

Thanks
S Rajkumar

By Anonymous, at 3:34 AM

• Hi,

Why winRunner does not support

.net Apllications

Thanks
S Rajkumar

By Rajkumar, at 2:16 AM

• Hi,

Are performance, load and stress test components of system test?

 Every organization is different. But I would assume the typical test life cycle is this. I
would break it down to these 4 major phases.
Unit Test --> Functional Test --> System Test -->Beta Test.
In the System Test phase, I would have these major tests: Integration Test,
Performance Test, Load Test, Stress Test and Scalability Test, etc.

Does anyone have any comment on my statement? Recently, we have some debate
within our Test Organization about the testing terminologies and methodologies. I
really want to know how other testers understand the test life cycle and definitions of
its components.

By Anonymous, at 12:23 PM

• http://www.testingreflections.com/node/view/2321

By Anonymous, at 9:35 PM

• Another cool tool for do performance testing is Eclipse project called Hyades.

With that you can easy test web applications, I really like to use it, its more simple
and more especific than OpenSta.

Also you can fin www.scapatech.com and see there a great tool (this is not a free
one)

By Alejandro Betancur, at 9:32 AM

• Hi all,
Is there any open soure tool for performance testing. If It there, Can support IE 6.0.
I heard abt open sta will support only 5.5.

By Anonymous, at 8:48 AM

• Thanks for your valuable discussion

Rajesh
 By Rajesh Duhan, at 3:44 AM

• Hi all
its realy a nice discussion
Rajesh Duhan

By Rajesh Duhan, at 3:46 AM

• How do "concurrent users" as set in a load test, relate to real traffic numbers? e.g. if
i want my site to cope with a peak traffic of 4000 hits/hour, what number of
concurrent hits should it be able to cope with? By my reckoning there is only one
answer: it has to cope with 4000 concurrent users, incase those 4000 hits/hour all
come at the same time. But then that's ridiculous!

By luca, at 7:23 AM

• Hi,

The explanation was great and upto the mark. I would like to know how to identify
Matrices(like bottle necks, performance of the appilcation). You can mail me at
ragu.karnam@gmail.com

Once again good one.....

Thanks
Raghu

By Raghavendra, at 6:30 AM

• Hi,
I am a gaduate student and my project is about web application testing tools. can
anybody let me know if Hyades creates a model from each web site and creates the
test cases from this model; or it duplicates the action sequences directly from weblog
or mimics recorded user actions?
 Thanks,
Anita

By Anita, at 11:04 PM

• I tried Neoload (published by Neotys) that is an easy to use load testing tool (but not
free).
Hope this helps.

By Anonymous, at 9:15 AM

• Hi,
I need to know what are the steps in a performance test.

Thanks

By Anonymous, at 4:33 AM

• Hi,

I am a load tester who hasn't got a project to work on yet.

My question is: how do I work like an expert, I mean what's the best methodology
that will impress my boss?
Remember your first load/perfomance project! How did you go about in preparing fo
rthe test?

Your contributions are more than wwelcome and please feel free to mail me at
randysage2001@yahoo.com

By Anonymous, at 2:07 AM
• just found this one...gives a good overview on some of the fundamental issues to
consider.
http://www.intel.com/cd/ids/developer/asmo-na/eng/257829.htm

By Anonymous, at 7:01 AM

• 1. double the baseline number for concurrent users/HTTP connections

randomly shut down and restart ports on the network switches/routers that connect
the servers (via SNMP commands for example) :
Here how it will justify that you have stress on system as the word itself say that you
have to give some stress on system and not the RECOVERY TIME, as this will
consider how much time the system will take to retain its stable working.

take the database offline, then restart it

Again this is recovery step...

run processes that consume resources (CPU, memory, disk, network) on the Web
and database servers
This step is correct but how will justify the statement where you have said "take
away resources ...". I am confused about these statement.
Please clear me.

By hemanshu, at 2:12 AM

• thanks for giving this information

By viji, at 1:52 AM

• i want Performance Testing by using JMeter, can any one help me

By naga, at 10:23 PM

• Performance Testing by using JMeter

 By naga, at 10:24 PM

• Ive been a performance and functional testing consultant for about 12 years now.
Here is the problem. Many people forget that performance and stress are not
interchangable. I dont think Grig couldve been more clearer.

The tools I use are LoadRunner, WinRunner, QuickTest Pro, Scapa, and others.

Right now I am on a project in Woking, UK, where the new system architechture is
being tested for performance, load and resilience. During this round of testing we are
loading the system with 5000 concurrent users performing normal user actions on
the site. The webservers, appservers and DB performance levels are constantly
monitored.

Now do you call this a performance test, load test or a stress test.

This scenario is a performance test. The system is being evaluated for its
performance under a constant load of users. They want to see if the proposed new
architechture is sufficient enough to support this load of users. However, this can be
classified as a load test as well, simply because there is a sustained load of users
(clients) hitting the system over a period of time.

Another scenario is this:

The system being tested is a new system architechture and tested with increasing
number of users to guage the performance of each component compared with the
load provided to the system.

This is a load test. Because there is no defined target and the only goal is to see
where the breaking point of the system is. This set of testing is not stress test
because no part of the system is in distress while the test is conducted.

Is everyone still unclear?

Michael Sky
 By Anonymous, at 9:10 AM

• From what I understand in the context of a website --

Performance Testing aims to answer (one of) the question(s) - How many concurrent
users can my site handle and still provide a n average response time to the user of
under 1 second?

Stress Testing aims to answer (one of) the question(s) - What would happen if the
database server crashes when hundreds of users are transacting online?

Am I thinking right? ;-)

By Siva, at 4:51 PM

• Hi Grig,
Has anyone come across a server-client testing tool, I mean not a web application.
Tool which could test application developed in Java.
But mainly I require that whatever test scenrios are developed should be shown in
Graphics and also the result should be mesureable.
Would be glad if anyone can provide details of such a tool.

By Priti, at 4:44 AM

• luca-
to replicate production test, just limit your resources in proportion to your users. So
if you want to test for 4000 sym. users in test, you look at your production machine
and half or quarter it. (ie. if production has a 2.7 Ghz box with 2 Gs of ram. In test
use a 1.3 Ghz machine with 1 G of ram. Then use 2000 users and so on.)

By Anonymous, at 12:02 PM

• its good but try to keep the difference in more simple words.
 By Anonymous, at 11:28 PM

• This seems focused on web testing. Any suggestions for tools to load test Windows
thick-client applications?

Thanks
daryl.cammack@us.ccl.com

By daryl.cammack@us.ccl.com, at 6:45 AM

• Daryl -- sorry, can't help you very much there, since I haven't used thick Windows
clients in a while....One approach would be to roll your own scripts that simulate key
presses, mouse events, etc.

Check out the tools (admittedly Python-based all of them) in the "GUI Testing"
section of the Python Testing Tools Taxonomy page I'm maintaining (I split the URL in
multiple lines, please join them together):

http://pycheesecake.org/
wiki/PythonTestingToolsTaxonomy

HTH,

Grig

By Grig Gheorghiu, at 7:26 AM

• Hi,
I want some of Test cases for load and performance testing for Webservices

By Anonymous, at 5:10 AM

• Hello Grig, your blog is very informative. Thanks a lot for providing such useful
information. But I still have a doubt, is load testing and volume testing the same? is
 there any specific difference between them.

I do not know weather it is true but i read somewhere that in volume testing we test
large sets of data using one user whereas in load testing large sets of data is tested
using multiple concurrent user.

Please suggest. waiting for your response

By Anonymous, at 2:30 AM

• Anonymous -- I don't think there is much difference between volume and load
testing. I would say though that load testing is usually applied to the system as a
whole, in order to see what its behavior is under constant load. Volume testing is
applied to specific sub-systems, to see how they react in the presence of high
volumes of data. But as with performance and load testing, these terms are
sometimes interchanged.

As far as I'm concerned, a sound testing strategy should incorporate these types of
testing, paying less attention to their names and more attention to actually running
them.

By Grig Gheorghiu, at 10:18 AM

• Thanks a lot, Grig.

By Vijay, at 5:20 AM

• Hi Grig,

I have to say that the information you have posted is very rich. I happen to have an
interview in London (City), and have really benefited from this Blogg of yours. I am
taking on a new role as a test analyst and this information has definately cleared
most of my doubts.

Could you please reply to my post. Regards.

 ola.alowooja@gmail.com

NB: Thanks everyone for your ideas and sharing you knowledge.

By Ola Alowooja - DORTCOME, at 6:31 PM

• thanks for your wonderful contexts

By Anonymous, at 6:58 PM

• Hi,

I am trying to figure out the differences between Loadrunner and ACT tools. Can any
of u guys help me?
Can u suggest which is a better tool?

Jeff.

By Anonymous, at 4:55 AM

• Hi Grig, i like very much your syntetic and clear explanation about difference in PLS
testing approach.
I translate in Italian your article to explain to non-technical users the difference
between these tests.
Because I see in it.wikipedia that this definition are absent, I would like to insert my
traslation from your article on it. I've seen you use CC license, but I'm not sure
about its limitation, can you give me your permission to public on Wikipedia? Thks in
advance, regards
Giovanni

By Anonymous, at 4:11 AM

• Giovanni -- sure, you have my permission to put your translation on Wikipedia.

Thanks!
 Grig

By Grig Gheorghiu, at 10:40 AM

• Good one

By swapna, at 11:04 PM

• Hey Grig,
like most of the readers (I think) I still don't see a clean line in the sand between the
three methodologies. What I think is clear is that depending on the situation, you
have to mix and match to get to where you want. If my understanding is clear, we
have 3 main metrics here:
. nominal performance
. throughput
. resilience
that can in turn be mapped to:
. performance testing (assessing a baseline)
. load testing (how many clients before performance degrades or, as customers
would ask, how many boxes do I need for my X customers)
. stress testing (what are the parameters that need to be monitored to make sure
the service remains available)

With that being said, I would like to come back to one of your first sentences:
"Ideally, the software under test is already stable..."

Performance is a very interesting metric to have even as the project just starts. It
will allow to pinpoint, as features get committed, which has a steep price in terms of
performance. Or, when the team focuses on performance improvements throughout
the project, it will allow to keep chaos under control by being able to quantify
improvements over baseline or over previous measurement.

Thank you for an interesting post that fosters and focuses the interest of the
community on one important aspect of testing.
 Arnaud

By Papa, at 6:45 PM

• Arnaud -- thanks for the thoughtful comments, your taxonomy makes sense. I agree
that doing performance testing early in the game has lots of benefis, but in my
experience very few people/teams do this.

Grig

By Grig Gheorghiu, at 9:19 AM

• Great article! I tried OpenSTA and Grinder, but found there was a lot of scripting and
debugging to get my tests to run properly as we have a very dynamic site.

I eventually came upon OpenLoad, which was much easier to use and relatively
inexpensive.

By Anonymous, at 1:13 PM

• what is the difference between stress testing and recovery testing?

By pavan giridhar, at 9:10 PM

• what is the difference between stress testing and recovery testing?

By pavan giridhar, at 9:11 PM

• I dont see any speicfic difference given in the post,it will be better if a comparision in
given between the three

By Arumugam, at 2:06 AM
• hi grig

a good overview.

as per my knowledge performance testing usually evaluates 3 things

1 throughput
2 response time and
3 latency

thanks & regards

murali

By Anonymous, at 11:35 PM

• I do not why people have a confusion between Performance and Stress.

Performance is you are veryfying the performance of an application for a set of users
i.e. say when 2000 users acess the system the system is serving them in 5 seconds.

Stress is the same above system is designed for 2000 users but there can be some
time in the business calendar when there will be 3000 users, as the application is
only designed for 2000 users the application can be slower than regular load may be
the system may take 9 seconds to serve a page, now if there are still more users say
1000 more that is total load is 4000, the system can run out of resources and may
crash.

Hope you got it

By Srinidhi, at 9:20 AM

• my name is Rekha Can anyone suggest me the goo book for testing which good
enough for the preparation of certification test in testing my id is
sarma.shruti@rediffmail.com
 By rekha, at 2:56 AM

• I must appreciate the way you have explained all three types of tests.
Being a non It , even I was able to understand with great ease.
Really worth explaining and wonderful explanation.
Thanks.

By Anonymous, at 11:57 PM

• The companies I worked as performance engineer have been doing the following
Performance tests:
Baseline/Benchmarks testing
Load/Stress load testing

In general:
Baseline/Benchmarks testing: need to be exec from build to builds, releases to
releases under the same hardware settings, the same test tools, the same workload
profiles, the same running time 10-30mins, collected the same required performance
matrices, mostly measuring the peak time product matrices, the purpose is to see if
the results fit product release requirement (such as web page latency < 3sec,
concurrent connections/users > 1k or 5k, %CPU usage < 85%), then comparing the
results between releases, build#, different venders' similar products. Most of time,
company will use this tests to collect the best numbers to advertise its product. Sale,
Marketing depts. are more interesting on those results. In real life, lots of time
people may never experience such good numbers. Such as mileages declared by
automobile company.

Load/Stress Load testing: we may use the same setup like benchmarks, but mostly
we used more complicated workload profiles, the test settings may be modified over
the time as long as the tests will be designed to excise the most of product
functionalities, the tests running with less intensive load, say if benchmark showing
the product can handle 2000 concurrent connections w/ latency less than 3-5sec, the
stress load with more complicated workload may only running 50-70% of 2000 conn.
But the load duration will be running from overnight about 12hrs to 72hrs or weeks.
Under the load, lots of functional tests may be performed. This kind of tests is more
 simulate real life scenarios.

During both above tests, we all collect product system resources usages by using
Win:PerfMon, Unix/Linux:vmstat/mpstat/iostat/netstat/lsof/ps/sar. %CPU, context
switch, memory, IO wait/queue, num of established/time-wait sockets are counters
helping to identify product’s bottlenecks & defects.

It seems like the different companies; even different projects within the same
company may have different process & methodologies to do their performance tests.
Performance release requirements & goals are very important as starting point.

By Anonymous, at 11:45 PM

• These comments have been invaluable to me as is this whole site. I thank you for
your comment.

By Rosie, at 10:02 AM

• Thanxs 4 good but explanation

I would like 2 add 1 more thing that
take ex of total tesing of chair==>
normal tresting ==if chair is dsnd 4 100 kg wt,n my wt is 70 kg then that teting is
called as nornal testing.
if my wt is 100 kg then that testing is called as load testing,
if my wt is 120 kg then that testing called as stress testing..

By Neelkanth, at 7:58 PM

• Hi Neelkanth,
Thanks for clear comments on Stress and load testing. But what about Performance
testing.

Thanks,
Rajiv
 By Rajiv Walia, at 10:56 PM

• Hi Grig,
Can we check Performance testing by using:

1. Load testing
2. Stress Testing

Right?

By Rajiv Walia, at 11:51 PM

• This post has been removed by the author.

By Roy, at 5:34 AM

• Actually I reccomend WebLOAD when talking about open source performance, stress
or load testing tools.

webload can be found here:

http://www.webload.org

By Anonymous, at 4:00 AM

• hi This is suman, i am new to laod runner, right now i need to test the web
application , same like the gmail(option) like

login-
Edit the mails
logout

can any one say what are the stpes to test this application

my email id is msuman75@gmail.com
 By msuman75, at 9:25 AM

• Very good compilation. Clearly and practically explains stress, load and performance
testing.

By Balaji, at 11:49 AM

• Good Article thanks.

Is there any open souce tool to test the network speed during the run of
performance other than the utilites like netstat etc. that comes with the OS?

By Simha, at 11:47 PM

• Simha -- check out iperf at http://dast.nlanr.net/Projects/Iperf/

Grig

By Grig Gheorghiu, at 7:34 AM

• large number of concurrent users, then the client machine will become the
bottleneck. At that point, you need to add more clients to your test and continue to
increase the load on the server.

Posted By Blackpool Hotels

Date: 19th September 2007

By Blackpool Hotels, at 1:37 AM

• Let me give you a scenario to help understand my question, There is a web intranet
application that is being accessed from different locations. When the users from the
Head Office are accessing this application they have no problem, however the remote
location users have a prblem of delayed response. The hardware on all the locations
including the HO is the same. How do I conclude if the problem is because of slow
network access or slow cpu processing. I have tried trace route and that indicates a
 mojor time lag between the hops for the problematic locations, however the HO
tracert is good. Also I have tried accessing a static page from all the locations and
the response time for this page is very big from the problematic location. Can I
conclude from this that the problem is with the network or is there anything else that
I can try. Very importantly, how can I clearly eliminate the possibility of slow cpu
response.

By Gokul Nadar, at 11:32 PM

• Gokul -- I think there's a simple test you can run to eliminate the possibility of a CPU
bottleneck. Run some utility on the server running the Web intranet application. If
the server is Unix/Linux-based, run top. If the server is Windows-based, run
Performance Monitor. Access your application from the HO and note the CPU
utilization. Access your app from a remote office and also note your CPU utilization.
My guess is that the 2 numbers will be almost identical. This means the bottleneck is
not CPU-bound, but, as you noted from the traceroutes, network-bound.

I'd also run iperf between the clients and the server -- it will tell you what kind of
network throughput you have on those links, and it will be another argument for the
network bottleneck. Iperf is available here: http://dast.nlanr.net/Projects/Iperf/