Ten names for load testing Load testing is a wide and established area of IT knowledge and software development

practices. There are many professionals who specialize here and testing gurus ready to provide useful advices and even teach you a theory on the subject. Surprisingly, the mentioned gurus often do not agree with each other on the very basic terms used in this field. If you search for information on load testing, most probably you will also find articles mentioning such terms as performance testing and stress testing. Are they all just synonyms? Everybody agrees that they are not, but still different sources provide different definitions for these terms. The most confusing point is the difference between performance and load testing. Some people reasonably say that since the performance of an application can be measured without creating any load on it, the load testing is a subset of performance testing. Other variants of performance testing may include measuring various parameters that do not depend on the load at all, such as time required to render a web page in the browser, or to perform any other action on the client side. When talking about stress testing, all agree that this is a type of testing when the server is stressed with a load above normal, and sometimes even beyond peak estimation for the tested application. However for what purpose is this done? Some say that this is just a way to check how the server responds to the rapidly growing load. In my own opinion such mess in terms is produced by marketing efforts of companies selling testing tools. They want to satisfy expectations of every potential customer coming to their web site. That is why they are providing similar descriptions for all three types of testing mentioned above. In other words, they do not want to lose customers who understand these terms differently. This would be really a dramatic loss taking in account that all the same tools are used for all types of load testing. Since I am not concentrated on selling anything to any particular customer right now, I have a freedom of developing a theory that would serve better understanding of the subject. So, no matter if any guru likes my classification, here it is. Load testing. I prefer to think of load testing as of a blanket term for all other types of testing that are done under the emulated load. Basically each of them can be described and distinguishing from other ones by specifying the following test options.

The main goal of test execution. The type and volume of applied load (it may be changing throughout the test). What parameters are measured and monitored when the test is performed.

Additional actions performed with the tested system during the test.

Performance testing. Here I consider the performance testing only as a type of load testing. I understand that it can have a wider meaning, but I want to mention only the performance testing done under the load. In this type of test we gradually increase the load by adding more and more virtual users to the test and check the performance parameters of the system at each test phase. The main things we monitor are:

Web site response time; Number of processed requests per second; Error rate.

As a result we have a graph showing performance parameters for each load level. So we can tell, for example, what response time we can expect under the estimated load. Since we also have the information on how it is changing throughout the test, we can also predict if this parameter can be improved by upgrading hardware and if it is stable.

Capacity testing. This type of test replies to the most common question in load testing: how many concurrent users the web site can handle while maintaining good response time and error rate. Again, we add virtual users gradually, but in this case we know the performance criteria in advance and just need to check that they are observed. When the performance starts to degrade significantly or just goes below our quality standard, we make the conclusion that the capacity limit is reached.

Stress testing. Every system has a capacity limit. When the load goes beyond it, the web site can start responding very slowly and even produce errors. The purposes of stress testing are:

Find that limit (in this respect it is similar to the capacity test); Check that when it is reached, the web site handle the stress correctly: produces graceful overload notifications and does not crash; When the load is reduced back to regular level, the web site returns to the normal operation retaining the performance parameters.

In my opinion it is very important to mention last two goals, because they show the specificity of stress testing.

Baseline testing. It is funny to write about that type of testing, but many people do this, so to make the list complete I have to mention it too. By baseline testing people understand some testing that is performed to establish standards for future tests. This is a bit strange, because I would recommend establishing such standards basing on your business requirements. Nevertheless I can imagine one case when such testing is really applicable. If you already have a live web site and you know that it is working more or less acceptable (you can have a good perception of it by checking cash in your pockets), you may perform baseline testing of that system to convert that perception to a more exact parameters, such as response time. After that you will be able to compare the performance of any new version of your web site with the initial data.

Endurance testing. This type of testing (also called soak testing) is used to check that the system can stand the load for a long time or a large number of transactions. It usually reveals various types of resource allocation problems. If a small memory leak is present in the system, it is not evident on a quick test, but will influence the performance after a long time. For endurance testing it is recommended to use changing periodic load to provoke resource reallocation. When the test is over we should compare resource usage and performance parameters on the early stages and at the end of test.

Volume testing. If your application can upload files, upload the largest ones. If it does the search, try to generate long results. Try to maximize the amount of processed data and the complexity of each transaction. This will be a volume testing. Note that in terms of the number of virtual users the load may remain on a regular estimated level throughout the test. We should already know the expected performance parameters for such load, so our goal is to check that they are not affected significantly by the above mentioned changes in the test data.

Spike testing. This is a test in which the load is increased and decreased very rapidly producing spikes. The goal is to check how the server responds to a very fast and significant load change. Depending on the web site implementation it may be reasonable to check this situation separately from other similar cases. For example, imagine a scalable system that can allocate additional resources when the load is increased. While it can work perfectly with the high target volume, it may

experience performance problems during resource allocation or just fail to do this correctly under such extreme load change.

Configuration testing. The goal of this type of testing is to find out how a change in software or hardware configuration affects the performance parameters under load. If your system consists of several components, you can try replacing some of them and see if the overall performance is changed. This way you can eliminate bottlenecks in your system and find optimal configuration.

Failover testing. This is a very interesting type of testing. It is performed under normal anticipated load for which we should already know the established performance parameters. After the initial warm-up phase we introduce an unexpected problem into the system. For example, it can be a connection problem between system components, which is easy to emulate by simply unplugging the network cable. We can also suddenly restart or disable redundant components without restarting the whole system. In this test we monitor two things.

How the performance parameters are affected by the introduced failure. What happens when the system comes back to normal conditions.

While it may be acceptable for the overall system performance to degrade temporary for a certain amount of time necessary to fix the failure, it is imperative that it is fully recovered after eliminating the problem. This is similar to the stress testing, but in this case the stress is produced not by the excessive load, but by a temporary problem inside the tested system. Well, this is the end of my list. One more thing that is worth mentioning is that the load testing in general is not completely separate and different from other testing practices. Some think that it is only reasonable to test how an application behaves under load at the very end of the development process just before it goes in production. This is not so. Of course, any load tests should be applied only after functional testing; otherwise the results will not be correct and useful. However you

can integrate various types of load tests into your regular development process and use them as part of regression testing performed on each new build or version of your web application.

Definition of Test
Being in the software industry, we have to encounter the word TEST many times. Though we have our own specific meaning of the word TEST, we have collected here some definitions of the word as provided by various dictionaries and other tidbits. The word TEST can be a Noun, a Verb or an Adjective but the definitions here are only of the Noun form. DEFINITION OF TEST

Google Dictionary: A Test is a deliberate action or experiment to find out how well something works. Cambridge Advanced Learners Dictionary: A Test is an act of using something to find out whether it is working correctly or how effective it is. The Free Dictionary by Farlex: A Test is a procedure for critical evaluation; a means of determining the presence, quality, or truth of something. Meriam Webster: A Test is a critical examination, observation, or evaluation. Dictionary.com: A Test is the means by which the presence, quality, or genuineness of anything is determined. WordWeb: A Test is trying something to find out about it. Longman Dictionary of Contemporary English: A Test is a process used to discover whether equipment or a product works correctly, or to discover more about it.

ETYMOLOGY OF TEST

Online Etymology Dictionary: TEST late 14c., small vessel used in assaying precious metals, from O.Fr. test, from L. testum earthen pot, related to testa piece of burned clay, earthen pot, shell (cf. L. testudo tortoise) and textere to weave (cf. Lith. tistas vessel made of willow twigs; see texture). Sense of trial or examination to determine the correctness of something is recorded from 1594. The verb in this sense is from 1748. The connecting notion is ascertaining the quality of a metal by melting it in a pot.
If the word TEST has been nauseating you because of its being overused, try the following synonyms:

Analysis Assessment Attempt Check Confirmation Evaluation Examination Experiment Inquiry Inspection Investigation Scrutiny Trial Verification

Software Testing Life Cycle

oftware Testing Life Cycle (STLC) defines the steps/stages/phases in testing of software. However, there is no fixed standard of STLC in the world and it basically varies as per the following:

Software Development Life Cycle Whims of the Management

Nevertheless, Software Testing Life Cycle, in general, comprises of the following phases:

Phase

Activity

Deliverables

Necessity

You review the software Requirements/Design requirements/design Review (Well, if they exist.) Once you have gathered a general idea of what needs to be tested, you plan for the tests. You design/detail your tests on the basis of detailed requirements/design of the software (sometimes, on the basis of your imagination). You setup the test environment

Review Defect Reports Test Plan Test Estimation Test Schedule

Curiosity

Test Planning

Farsightedness

Test Designing

Test Cases / Test Scripts /Test Data Requirements Traceability Matrix

Creativity

Test Environment Setup

Test Environment Rich company

(server/client/network, etc) with the goal of replicating the endusers environment. You execute your Test Cases/Scripts in the Test Environment to see whether they pass.

Test Execution

Test Results (Incremental) Defect Reports Test Results (Final) Test/Defect Metrics Test Closure Report Who Worked Till Late & on Weekends Report

Patience

Test Reporting

You prepare various reports for various stakeholders.

Diplomacy

Interestingly, no matter how well-defined a Software Testing Life Cycle you have in your project or organization, there are chances that you will invariably witness the following widely-popular cycle:

Testing Cursing

In this type of STLC, you skip phases like requirement/design review, test planning, and test designing in the high hope that the skipping will save you some time and/or cost. Also, note that the Software Testing Life Cycle phases mentioned above do not necessarily have to be in the order listed; some phases can sometimes run in parallel (For instance, Test Designing and Test Execution). And, in extreme cases, the phases might also be reversed (For instance, when there is Cursing prior to Testing).

Software Development Life Cycle

SOFTWARE DEVELOPMENT LIFE CYCLE [SDLC] Information: Software Development Life Cycle, or Software Development Process, defines the steps/stages/phases in the building of software. There are various kinds of software development models like:

Waterfall model Spiral model Iterative and incremental development (like Unified Process and Rational Unified Process) Agile development (like Extreme Programming and Scrum)

Models are evolving with time and the development life cycle can vary significantly from one model to the other. It is beyond the scope of this particular article to discuss each model. However, each model comprises of all or some of the following phases/activities/tasks. SDLC IN SUMMARY

Project Planning Requirements Development Estimation Scheduling Design Coding Test Build/Deployment Unit Testing Integration Testing User Documentation System Testing Acceptance Testing Production Build/Deployment Release Maintenance

SDLC IN DETAIL

Project Planning o Prepare o Review o Rework o Baseline o Revise [if necessary] >> Review >> Rework >> Baseline Requirements Development[Business Requirements and Software/Product Requirements] o Develop o Review o Rework o Baseline o Revise [if necessary] >> Review >> Rework >> Baseline

Estimation[Size / Effort / Cost] o <same as the activities/tasks mentioned for Project Planning> Scheduling o <same as the activities/tasks mentioned for Project Planning> Designing[ High Level Design and Detail Design] o <same as the activities/tasks mentioned for Requirements Development> Coding o Code o Review o Rework o Commit o Recode [if necessary] >> Review >> Rework >> Commit Test Builds Preparation/Deployment o Build/Deployment Plan Prepare Review Rework Baseline Revise [if necessary] >> Review >> Rework >> Baseline o Build/Deploy Unit Testing o Test Plan Prepare Review Rework Baseline Revise [if necessary] >> Review >> Rework >> Baseline o Test Cases/Scripts Prepare Review Rework Baseline Execute Revise [if necessary] >> Review >> Rework >> Baseline >> Execute Integration Testing o <same as the activities/tasks mentioned for unit testing> User Documentation o Prepare o Review o Rework o Baseline o Revise [if necessary] >> Review >> Rework >> Baseline

System Testing o <same as the activities/tasks mentioned for Unit Testing> Acceptance Testing[ Internal Acceptance Test and External Acceptance Test] o <same as the activities/tasks mentioned for Unit Testing> Production Build/Deployment o <same as the activities/tasks mentioned for Test Build/Deployment> Release o Prepare o Review o Rework o Release Maintenance o Recode [Enhance software / Fix bugs] o Retest o Redeploy o Rerelease

Notes:

The life cycle mentioned here is NOT set in stone and each phase does not necessarily have to be implemented in the order mentioned. Though SDLC uses the term Development, it does not focus just on the coding tasks done by developers but incorporates the tasks of all stakeholders, including testers.

There may still be many other activities/ tasks which have not been specifically mentioned above, like Configuration Management. No matter what, it is essential that you clearly understand the software development life cycle your project is following. One issue that is widespread in many projects is that software testers are involved much later in the life cycle, due to which they lack visibility and authority (which ultimately compromises software quality).

Test Plan
TEST PLAN DEFINITION A Software Test Plan is a document describing the testing scope and activities. It is the basis for formally testing any software/product in a project.

ISTQB Definition

test plan: A document describing the scope, approach, resources and schedule of intended test activities. It identifies amongst others test items, the features to be tested, the testing tasks, who will do each task, degree of tester independence, the test environment, the test design techniques and entry and exit criteria to be used, and the rationale for their choice,and any risks requiring contingency planning. It is a record of the test planning process. master test plan: A test plan that typically addresses multiple test levels. phase test plan: A test plan that typically addresses one test phase.

TEST PLAN TYPES One can have the following types of test plans:

Master Test Plan: A single high-level test plan for a project/product that unifies all other test plans. Testing Level Specific Test Plans:Plans for each level of testing. o Unit Test Plan o Integration Test Plan

System Test Plan Acceptance Test Plan Testing Type Specific Test Plans: Plans for major types of testing like Performance Test Plan and Security Test Plan.
o o

TEST PLAN TEMPLATE

The format and content of a software test plan vary depending on the processes, standards, and test management tools being implemented. Nevertheless, the following format, which is based on IEEE standard for software test documentation, provides a summary of what a test plan can/should contain.
Test Plan Identifier:

Provide a unique identifier for the document. (Adhere to the Configuration Management System if you have one.)

Introduction:

Provide an overview of the test plan. Specify the goals/objectives. Specify any constraints.

References:

List the related documents, with links to them if available, including the following: o Project Plan o Configuration Management Plan

Test Items:

List the test items (software/products) and their versions.

Features to be Tested:

List the features of the software/product to be tested. Provide references to the Requirements and/or Design specifications of the features to be tested

Features Not to Be Tested:

List the features of the software/product which will not be tested. Specify the reasons these features wont be tested.

Approach:

Mention the overall approach to testing. Specify the testing levels [if it's a Master Test Plan], the testing types, and the testing methods [Manual/Automated; White Box/Black Box/Gray Box]

Item Pass/Fail Criteria:

Specify the criteria that will be used to determine whether each test item (software/product) has passed or failed testing.

Suspension Criteria and Resumption Requirements:

Specify criteria to be used to suspend the testing activity. Specify testing activities which must be redone when testing is resumed.

Test Deliverables:

List test deliverables, and links to them if available, including the following: o Test Plan (this document itself) o Test Cases o Test Scripts o Defect/Enhancement Logs o Test Reports

Test Environment:

Specify the properties of test environment: hardware, software, network etc. List any testing or related tools.

Estimate:

Provide a summary of test estimates (cost or effort) and/or provide a link to the detailed estimation.

Schedule:

Provide a summary of the schedule, specifying key test milestones, and/or provide a link to the detailed schedule.

Staffing and Training Needs:

Specify staffing needs by role and required skills.

Identify training that is necessary to provide those skills, if not already acquired.

Responsibilities:

List the responsibilities of each team/role/individual.

Risks:

List the risks that have been identified. Specify the mitigation plan and the contingency plan for each risk.

Assumptions and Dependencies:

List the assumptions that have been made during the preparation of this plan. List the dependencies.

Approvals:

Specify the names and roles of all persons who must approve the plan. Provide space for signatures and dates. (If the document is to be printed.)

TEST PLAN GUIDELINES

Make the plan concise. Avoid redundancy and superfluousness. If you think you do not need a section that has been mentioned in the template above, go ahead and delete that section in your test plan. Be specific. For example, when you specify an operating system as a property of a test environment, mention the OS Edition/Version as well, not just the OS Name. Make use of lists and tables wherever possible. Avoid lengthy paragraphs. Have the test plan reviewed a number of times prior to baselining it or sending it for approval. The quality of your test plan speaks volumes about the quality of the testing you or your team is going to perform. Update the plan as and when necessary. An out-dated and unused document stinks and is worse than not having the document in the first place.

Test Case
DEFINITION A test case is a set of conditions or variables under which a tester will determine whether a system under test satisfies requirements or works correctly.

The process of developing test cases can also help find problems in the requirements or design of an application. TEST CASE TEMPLATE A test case can have the following elements. Note, however, that normally a test management tool is used by companies and the format is determined by the tool used. Test Suite ID Test Case ID Test Case Summary Related Requirement The ID of the test suite to which this test case belongs. The ID of the test case.

The summary / objective of the test case.

The ID of the requirement this test case relates/traces to.

Prerequisites

Any prerequisites or preconditions that must be fulfilled prior to executing the test. Step-by-step procedure to execute the test. The test data, or links to the test data, that are to be used while conducting the test.

Test Procedure

Test Data

Expected Result The expected result of the test. Actual Result The actual result of the test; to be filled after executing the test. Pass or Fail. Other statuses can be Not Executed if testing is not performed and Blocked if testing is blocked. Any comments on the test case or test execution. The name of the author of the test case.

Status

Remarks Created By

Date of Creation The date of creation of the test case. Executed By Date of Execution Test Environment The name of the person who executed the test.

The date of execution of the test.

The environment (Hardware/Software/Network) in which the test was executed.

TEST CASE EXAMPLE / TEST CASE SAMPLE Test Suite ID Test Case ID Test Case Summary Related Requirement TS001 TC001

To verify that clicking the Generate Coin button generates coins.

RS001 1. User is authorized. 2. Coin balance is available. 1. Select the coin denomination in the Denomination field. 2. Enter the number of coins in the Quantity field. 3. Click Generate Coin.

Prerequisites

Test Procedure

Test Data Expected Result

1. Denominations: 0.05, 0.10, 0.25, 0.50, 1, 2, 5 2. Quantities: 0, 1, 5, 10, 20

1. Coin of the specified denomination should be produced if the

specified Quantity is valid (1, 5) 2. A message Please enter a valid quantity between 1 and 10

should be displayed if the specified quantity is invalid. 1. If the specified quantity is valid, the result is as expected. Actual Result
2. If the specified quantity is invalid, nothing happens; the expected

message is not displayed Status Remarks Created By Fail This is a sample test case. John Doe

Date of Creation 01/14/2020 Executed By Date of Execution Test Environment Jane Roe 02/16/2020

OS: Windows Y Browser: Chrome N

WRITING GOOD TEST CASES

As far as possible, write test cases in such a way that you test only one thing at a time. Do not overlap or complicate test cases. Attempt to make your test cases atomic. Ensure that all positive scenarios and negative scenarios are covered. Language: o Write in simple and easy to understand language. o Use active voice: Do this, do that. o Use exact and consistent names (of forms, fields, etc). Characteristics of a good test case: o Accurate: Exacts the purpose. o Economical: No unnecessary steps or words. o Traceable: Capable of being traced to requirements. o Repeatable: Can be used to perform the test over and over. o Reusable: Can be reused if necessary.

Test Script

A Test Script is a set of instructions (written using a scripting/programming language) that is performed on a system under test to verify that the system performs as expected. Test scripts are used in automated testing. Sometimes, a set of instructions (written in a human language), used in manual testing, is also called a Test Script but a better term for that would be a Test Case. Some scripting languages used in automated testing are:

JavaScript Perl Python Ruby Tcl Unix Shell Script VBScript

There are also many Test Automation Tools/Frameworks that generate the test scripts for you; without the need for actual coding. Many of these tools have their own scripting languages (some of them based on a core scripting languages). For example, Sikuli, a GUI automation tool, uses Sikuli Script which is based on Python. A test script can be as simple as the one below: def sample_test_script (self): type ("TextA") click (ImageButtonA) assertExist (ImageResultA) A test execution engine and a repository of test scripts (along with test data) are collectively known as a Test Harness.

Black Box Testing

DEFINITION Black Box Testing, also known as Behavioral Testing, is a software testing method in which the internal structure/design/implementation of the item being tested is not known to the tester. These tests can be functional or non-functional, though usually functional.

This method is named so because the software program, in the eyes of the tester, is like a black box; inside which one cannot see. Black Box Testing is contrasted with White Box Testing. View Differences between Black Box Testing and White Box Testing. This method of attempts to find errors in the following categories:

Incorrect or missing functions Interface errors Errors in data structures or external database access Behavior or performance errors Initialization and termination errors

EXAMPLE A tester, without knowledge of the internal structures of a website, tests the web pages by using a browser; providing inputs (clicks, keystrokes) and verifying the outputs against the expected outcome. LEVELS APPLICABLE TO Black Box Testing method is applicable to all levels of the software testing process:

Unit Testing Integration Testing System Testing Acceptance Testing

The higher the level, and hence the bigger and more complex the box, the more black box testing method comes into use. BLACK BOX TESTING TECHNIQUES Following are some techniques that can be used for designing black box tests.

Equivalence partitioning

Equivalence Partitioning is a software test design technique that involves dividing input values into valid and invalid partitions and selecting representative values from each partition as test data.

Boundary Value Analysis

Boundary Value Analysis is a software test design technique that involves determination of boundaries for input values and selecting values that are at the boundaries and just inside/outside of the boundaries as test data.

Cause Effect Graphing

Cause Effect Graphing is a software test design technique that involves identifying the cases (input conditions) and effects (output conditions), producing a Cause-Effect Graph, and generating test cases accordingly. BLACK BOX TESTING ADVANTAGES

Tests are done from a users point of view and will help in exposing discrepancies in the specifications Tester need not know programming languages or how the software has been implemented Tests can be conducted by a body independent from the developers, allowing for an objective perspective and the avoidance of developer-bias Test cases can be designed as soon as the specifications are complete

BLACK BOX TESTING DISADVANTAGES

Only a small number of possible inputs can be tested and many program paths will be left untested Without clear specifications, which is the situation in many projects, test cases will be difficult to design Tests can be redundant if the software designer/ developer has already run a test case. Ever wondered why a soothsayer closes the eyes when foretelling events? So is almost the case in Black Box Testing.

Definition by ISTQB

black box testing: Testing, either functional or non-functional, without reference to the internal structure of the component or system. black box test design technique: Procedure to derive and/or select test cases based on an analysis of the specification, either functional or non-functional, of a component or

system without reference to its internal structure.

Test Case
DEFINITION A test case is a set of conditions or variables under which a tester will determine whether a system under test satisfies requirements or works correctly. The process of developing test cases can also help find problems in the requirements or design of an application. TEST CASE TEMPLATE A test case can have the following elements. Note, however, that normally a test management tool is used by companies and the format is determined by the tool used. Test Suite ID Test Case ID Test Case Summary Related Requirement The ID of the test suite to which this test case belongs. The ID of the test case.

The summary / objective of the test case.

The ID of the requirement this test case relates/traces to.

Prerequisites

Any prerequisites or preconditions that must be fulfilled prior to executing the test. Step-by-step procedure to execute the test. The test data, or links to the test data, that are to be used while conducting the test.

Test Procedure

Test Data

Expected Result The expected result of the test. Actual Result The actual result of the test; to be filled after executing the test. Pass or Fail. Other statuses can be Not Executed if testing is not performed and Blocked if testing is blocked. Any comments on the test case or test execution. The name of the author of the test case.

Status

Remarks Created By

Date of Creation The date of creation of the test case. Executed By Date of Execution Test Environment The name of the person who executed the test.

The date of execution of the test.

The environment (Hardware/Software/Network) in which the test was executed.

TEST CASE EXAMPLE / TEST CASE SAMPLE Test Suite ID Test Case ID Test Case Summary Related Requirement Prerequisites TS001 TC001

To verify that clicking the Generate Coin button generates coins.

RS001 1. User is authorized.

2. Coin balance is available. 1. Select the coin denomination in the Denomination field. 2. Enter the number of coins in the Quantity field. 3. Click Generate Coin. Test Data 1. Denominations: 0.05, 0.10, 0.25, 0.50, 1, 2, 5 2. Quantities: 0, 1, 5, 10, 20 1. Coin of the specified denomination should be produced if the specified Quantity is valid (1, 5)
2. A message Please enter a valid quantity between 1 and 10

Test Procedure

Expected Result

should be displayed if the specified quantity is invalid. 1. If the specified quantity is valid, the result is as expected. Actual Result
2. If the specified quantity is invalid, nothing happens; the expected

message is not displayed Status Remarks Created By Fail This is a sample test case. John Doe

Date of Creation 01/14/2020 Executed By Date of Execution Test Environment Jane Roe 02/16/2020

OS: Windows Y Browser: Chrome N

WRITING GOOD TEST CASES

As far as possible, write test cases in such a way that you test only one thing at a time. Do not overlap or complicate test cases. Attempt to make your test cases atomic. Ensure that all positive scenarios and negative scenarios are covered. Language: o Write in simple and easy to understand language.

Use active voice: Do this, do that. Use exact and consistent names (of forms, fields, etc). Characteristics of a good test case: o Accurate: Exacts the purpose. o Economical: No unnecessary steps or words. o Traceable: Capable of being traced to requirements. o Repeatable: Can be used to perform the test over and over. o Reusable: Can be reused if necessary.
o o

Unit Testing
DEFINITION Unit Testing is a level of the software testing process where individual units/components of a software/system are tested. The purpose is to validate that each unit of the software performs as designed.

A unit is the smallest testable part of software. It usually has one or a few inputs and usually a single output. In procedural programming a unit may be an individual program, function, procedure, etc. In object-oriented programming, the smallest unit is a method, which may belong to a base/super class, abstract class or derived/child class. (Some treat a module of an application as a unit. This is to be discouraged as there will probably be many individual units within that module.) Unit testing frameworks, drivers, stubs and mock or fake objects are used to assist in unit testing. METHOD

Unit Testing is performed by using the White Box Testing method. When is it performed? Unit Testing is the first level of testing and is performed prior to Integration Testing. Who performs it? Unit Testing is normally performed by software developers themselves or their peers. In rare cases it may also be performed by independent software testers. TASKS

Unit Test Plan o Prepare o Review o Rework o Baseline Unit Test Cases/Scripts o Prepare o Review o Rework o Baseline Unit Test o Perform

BENEFITS

Unit testing increases confidence in changing/maintaining code. If good unit tests are written and if they are run every time any code is changed, the likelihood of any defects due to the change being promptly caught is very high. If unit testing is not in place, the most one can do is hope for the best and wait till the test results at higher levels of testing are out. Also, if codes are already made less interdependent to make unit testing possible, the unintended impact of changes to any code is less. Codes are more reusable. In order to make unit testing possible, codes need to be modular. This means that codes are easier to reuse. Development is faster. How? If you do not have unit testing in place, you write your code and perform that fuzzy developer test (You set some breakpoints, fire up the GUI, provide a few inputs that hopefully hit your code and hope that you are all set.) In case you have unit testing in place, you write the test, code and run the tests. Writing tests takes time but the time is compensated by the time it takes to run the tests. The test runs take very less time: You need not fire up the GUI and provide all those inputs. And, of course, unit tests are more reliable than developer tests. Development is faster in the long run too. How? The effort

required to find and fix defects found during unit testing is peanuts in comparison to those found during system testing or acceptance testing. The cost of fixing a defect detected during unit testing is lesser in comparison to that of defects detected at higher levels. Compare the cost (time, effort, destruction, humiliation) of a defect detected during acceptance testing or say when the software is live. Debugging is easy. When a test fails, only the latest changes need to be debugged. With testing at higher levels, changes made over the span of several days/weeks/months need to be debugged. Codes are more reliable. Why? I think there is no need to explain this to a sane person.

TIPS

Find a tool/framework for your language. Do not create test cases for everything: some will be handled by themselves. Instead, focus on the tests that impact the behavior of the system. Isolate the development environment from the test environment. Use test data that is close to that of production. Before fixing a defect, write a test that exposes the defect. Why? First, you will later be able to catch the defect if you do not fix it properly. Second, your test suite is now more comprehensive. Third, you will most probably be too lazy to write the test after you have already fixed the defect. Write test cases that are independent of each other. For example if a class depends on a database, do not write a case that interacts with the database to test the class. Instead, create an abstract interface around that database connection and implement that interface with mock object. Aim at covering all paths through the unit. Pay particular attention to loop conditions. Make sure you are using a version control system to keep track of your code as well as your test cases. In addition to writing cases to verify the behavior, write cases to ensure performance of the code. Perform unit tests continuously and frequently.

ONE MORE REASON Lets say you have a program comprising of two units. The only test you perform is system testing. [You skip unit and integration testing.] During testing, you find a bug. Now, how will you determine the cause of the problem?

Is Is Is Is

the bug due to an error in unit 1? the bug due to an error in unit 2? the bug due to errors in both units? the bug due to an error in the interface between the units?

Is the bug due to an error in the test or test case?

Unit testing is often neglected but it is, in fact, the most important level of testing. Definition by ISTQB

unit testing: See component testing. component testing: The testing of individual software components.

Integration Testing
DEFINITION Integration Testing is a level of the software testing process where individual units are combined and tested as a group.

The purpose of this level of testing is to expose faults in the interaction between integrated units. Test drivers and test stubs are used to assist in Integration Testing.

Note: The definition of a unit is debatable and it could mean any of the following:
1. the smallest testable part of a software 2. a module which could consist of many of 1 3. a component which could consist of many of 2 ANALOGY

During the process of manufacturing a ballpoint pen, the cap, the body, the tail and clip, the ink cartridge and the ballpoint are produced separately and unit tested separately. When two or more units are ready, they are assembled and Integration Testing is performed. For example, whether the cap fits into the body or not. METHOD Any of Black Box Testing, White Box Testing, and Gray Box Testing methods can be used. Normally, the method depends on your definition of unit. TASKS

Integration Test Plan o Prepare o Review o Rework o Baseline Integration Test Cases/Scripts o Prepare o Review o Rework o Baseline Integration Test o Perform

When is Integration Testing performed? Integration Testing is performed after Unit Testing and before System Testing. Who performs Integration Testing? Either Developers themselves or independent Testers perform Integration Testing. APPROACHES

Big Bang is an approach to Integration Testing where all or most of the units are combined together and tested at one go. This approach is taken when the testing team receives the entire software in a bundle. So what is the difference between Big Bang Integration Testing and System Testing? Well, the former tests only the interactions between the units while the latter tests the entire system. Top Down is an approach to Integration Testing where top level units are tested first and lower level units are tested step by step after that. This approach is taken when top down development approach is followed. Test Stubs are needed to simulate lower level units which may not be available during the initial phases.

Bottom Up is an approach to Integration Testing where bottom level units are tested first and upper level units step by step after that. This approach is taken when bottom up development approach is followed. Test Drivers are needed to simulate higher level units which may not be available during the initial phases. Sandwich/Hybrid is an approach to Integration Testing which is a combination of Top Down and Bottom Up approaches.

TIPS

Ensure that you have a proper Detail Design document where interactions between each unit are clearly defined. In fact, you will not be able to perform Integration Testing without this information. Ensure that you have a robust Software Configuration Management system in place. Or else, you will have a tough time tracking the right version of each unit, especially if the number of units to be integrated is huge. Make sure that each unit is first unit tested before you start Integration Testing. As far as possible, automate your tests, especially when you use the Top Down or Bottom Up approach, since regression testing is important each time you integrate a unit, and manual regression testing can be inefficient.

Definition by ISTQB

integration testing: Testing performed to expose defects in the interfaces and in the interactions between integrated components or systems. See also component integration testing, system integration testing. component integration testing: Testing performed to expose defects in the interfaces and interaction between integrated components. system integration testing: Testing the integration of systems and packages; testing interfaces to external organizations (e.g. Electronic Data Interchange, Internet).

System Testing
DEFINITION System Testing is a level of the software testing process where a complete, integrated system/software is tested. The purpose of this test is to evaluate the systems compliance with the specified requirements.

ANALOGY During the process of manufacturing a ballpoint pen, the cap, the body, the tail, the ink cartridge and the ballpoint are produced separately and unit tested separately. When two or more units are ready, they are assembled and Integration Testing is performed. When the complete pen is integrated, System Testing is performed. METHOD Usually, Black Box Testing method is used. TASKS

System Test Plan o Prepare o Review o Rework o Baseline System Test Cases o Prepare o Review o Rework o Baseline System Test o Perform

When is it performed? System Testing is performed after Integration Testing and before Acceptance Testing.

Who performs it? Normally, independent Testers perform System Testing. Definition by ISTQB

system testing: The process of testing an integrated system to verify that it meets specified

Acceptance Testing
DEFINITION Acceptance Testing is a level of the software testing process where a system is tested for acceptability. The purpose of this test is to evaluate the systems compliance with the business requirements and assess whether it is acceptable for delivery.

ANALOGY During the process of manufacturing a ballpoint pen, the cap, the body, the tail and clip, the ink cartridge and the ballpoint are produced separately and unit tested separately. When two or more units are ready, they are assembled and Integration Testing is performed. When the complete pen is integrated, System Testing is performed. Once the System Testing is complete, Acceptance Testing is performed so as to confirm that the ballpoint pen is ready to be made available to the end-users. METHOD

Usually, Black Box Testing method is used in Acceptance Testing. Testing does not usually follow a strict procedure and is not scripted but is rather ad-hoc. TASKS

Acceptance Test Plan o Prepare o Review o Rework o Baseline Acceptance Test Cases/Checklist o Prepare o Review o Rework o Baseline Acceptance Test o Perform

When is it performed? Acceptance Testing is performed after System Testing and before making the system available for actual use. Who performs it?

Internal Acceptance Testing (Also known as Alpha Testing) is performed by members of the organization that developed the software but who are not directly involved in the project (Development or Testing). Usually, it is the members of Product Management, Sales and/or Customer Support. External Acceptance Testing is performed by people who are not employees of the organization that developed the software. o Customer Acceptance Testing is performed by the customers of the organization that developed the software. They are the ones who asked the organization to develop the software for them. [This is in the case of the software not being owned by the organization that developed it.] o User Acceptance Testing (Also known as Beta Testing) is performed by the end users of the software. They can be the customers themselves or the customers customers.

Definition by ISTQB

acceptance testing: Formal testing with respect to user needs, requirements, and business processes conducted to determine whether or not a system satisfies

the acceptance criteria and to enable the user, customers or other authorized entity to determine whether or not to accept the system. MORE

Defect
DEFINITION A Software Defect / Bug is a condition in a software product which does not meet a software requirement (as stated in the requirement specifications) or end-user expectations (which may not be specified but are reasonable). In other words, a defect is an error in coding or logic that causes a program to malfunction or to produce incorrect/unexpected results.

A program that contains a large number of bugs is said to be buggy. Reports detailing bugs in software are known as bug reports. (See Defect Report) Applications for tracking bugs are known as bug tracking tools. The process of finding the cause of bugs is known as debugging. The process of intentionally injecting bugs in a software program, to estimate test coverage by monitoring the detection of those bugs, is known as bebugging.

Software Testing proves that defects exist but NOT that defects do not exist. CLASSIFICATION Software Defects/ Bugs are normally classified as per:

Severity / Impact (See Defect Severity)

Probability / Visibility (See Defect Probability) Priority / Urgency (See Defect Priority) Related Dimension of Quality (See Dimensions of Quality) Related Module / Component Phase Detected Phase Injected

Related Module /Component Related Module / Component indicates the module or component of the software where the defect was detected. This provides information on which module / component is buggy or risky.

Module/Component A Module/Component B Module/Component C

Phase Detected Phase Detected indicates the phase in the software development lifecycle where the defect was identified.

Unit Testing Integration Testing System Testing Acceptance Testing

Phase Injected Phase Injected indicates the phase in the software development lifecycle where the bug was introduced. Phase Injected is always earlier in the software development lifecycle than the Phase Detected. Phase Injected can be known only after a proper root-cause analysis of the bug.

Requirements Development High Level Design Detailed Design Coding Build/Deployment

Note that the categorizations above are just guidelines and it is up to the project/organization to decide on what kind of categorization to use. In most cases, the categorization depends on the defect tracking tool that is being used. It is essential that

project members agree beforehand on the categorization (and the meaning of each categorization) so as to avoid arguments, conflicts, and unhealthy bickering later. NOTE: We prefer the term Defect over the term Bug because Defect is more comprehensive.

Defect Severity
Defect Severity or Impact is a classification of software defect (bug) to indicate the degree of negative impact on the quality of software. ISTQB Definition

severity: The degree of impact that a defect has on the development or operation of a component or system.

DEFECT SEVERITY CLASSIFICATION The actual terminologies, and their meaning, can vary depending on people, projects, organizations, or defect tracking tools, but the following is a normally accepted classification.

Critical: The defect affects critical functionality or critical data. It does not have a workaround. Example: Unsuccessful installation, complete failure of a feature. Major: The defect affects major functionality or major data. It has a workaround but is not obvious and is difficult. Example: A feature is not functional from one module but the task is doable if 10 complicated indirect steps are followed in another module/s. Minor: The defect affects minor functionality or non-critical data. It has an easy workaround. Example: A minor feature that is not functional in one module but the same task is easily doable from another module. Trivial: The defect does not affect functionality or data. It does not even need a workaround. It does not impact productivity or efficiency. It is merely an inconvenience. Example: Petty layout discrepancies, spelling/grammatical errors.

Severity is also denoted as:

S1 S2 S3 S4

= Critical = Major = Minor = Trivial

CAUTION:

Defect Severity is one of the most common causes of feuds between Testers and Developers. A typical situation is where a Tester classifies the Severity of Defect as Critical or Major but the Developer refuses to accept that: He/she believes that the defect is of Minor or Trivial severity. Though we have provided you some guidelines in this article on how to interpret each level of severity, this is still a very subjective matter and chances are high that one will not agree with the definition of the other. You can however lessen the chances of differing opinions in your project by discussing (and documenting, if necessary) what each level of severity means and by agreeing to at least some standards (substantiating with examples, if necessary.) ADVICE: Go easy on this touchy defect dimension and good luck!

Defect Priority
Defect Priority (Bug Priority) indicates the importance or urgency of fixing a defect. Though priority may be initially set by the Software Tester, it is usually finalized by the Project/Product Manager. Priority can be categorized into the following levels:

Urgent: Must be fixed in the next build. High: Must be fixed in any of the upcoming builds but should be included in the release. Medium: May be fixed after the release / in the next release. Low: May or may not be fixed at all.

Priority is also denoted as P1 for Urgent, P2 for High and so on. NOTE: Priority is quite a subjective decision; do not take the categorizations above as authoritative. However, at a high level, priority is determined by considering the following:

Business need for fixing the defect Severity/Impact Probability/Visibility Available Resources (Developers to fix and Testers to verify the fixes) Available Time (Time for fixing, verifying the fixes and performing regression tests after the verification of the fixes)

ISTQB Definition:

priority: The level of (business) importance assigned to an item, e.g. defect.

Defect Priority needs to be managed carefully in order to avoid product instability, especially when there is a large of number of defects.

Defect Life Cycle

Defect Life Cycle (Bug Life cycle) is the journey of a defect from its identification to its closure. The Life Cycle varies from organization to organization and is governed by the software testing process the organization or project follows and/or the Defect tracking tool being used. Nevertheless, the life cycle in general resembles the following:

Status NEW

Alternative Status

ASSIGNED DEFERRED DROPPED COMPLETED REASSIGNED CLOSED Defect Status Explanation

OPEN

REJECTED FIXED, RESOLVED, TEST REOPENED VERIFIED

NEW: Tester finds a defect and posts it with the status NEW. This defect is yet to be studied/approved. The fate of a NEW defect is one of ASSIGNED, DROPPED and DEFERRED. ASSIGNED / OPEN: Test / Development / Project lead studies the NEW defect and if it is found to be valid it is assigned to a member of the Development Team. The assigned Developers responsibility is now to fix the defect and have it COMPLETED. Sometimes, ASSIGNED and OPEN can be different statuses. In that case, a defect can be open yet unassigned. DEFERRED: If a valid NEW or ASSIGNED defect is decided to be fixed in upcoming releases instead of the current release it is DEFERRED. This defect is ASSIGNED when the time comes. DROPPED / REJECTED: Test / Development/ Project lead studies the NEW defect and if it is found to be invalid, it is DROPPED / REJECTED. Note that the specific reason for this action needs to be given. COMPLETED / FIXED / RESOLVED / TEST: Developer fixes the defect that is ASSIGNED to him or her. Now, the fixed defect needs to be verified by the Test Team and the Development Team assigns the defect back to the Test Team. A COMPLETED defect is either CLOSED, if fine, or REASSIGNED, if still not fine. If a Developer cannot fix a defect, some organizations may offer the following statuses: o Wont Fix / Cant Fix: The Developer will not or cannot fix the defect due to some reason. o Cant Reproduce: The Developer is unable to reproduce the defect. o Need More Information: The Developer needs more information on the defect from the Tester.

REASSIGNED / REOPENED: If the Tester finds that the fixed defect is in fact not fixed or only partially fixed, it is reassigned to the Developer who fixed it. A REASSIGNED defect needs to be COMPLETED again. CLOSED / VERIFIED: If the Tester / Test Lead finds that the defect is indeed fixed and is no more of any concern, it is CLOSED / VERIFIED. This is the happy ending.

Defect Life Cycle Implementation Guidelines

Make sure the entire team understands what each defect status exactly means. Also, make sure the defect life cycle is documented. Ensure that each individual clearly understands his/her responsibility as regards each defect. Ensure that enough detail is entered in each status change. For example, do not simply DROP a defect but provide a reason for doing so. If a defect tracking tool is being used, avoid entertaining any defect related requests without an appropriate change in the status of the defect in the tool. Do not let anybody take shortcuts. Or else, you will never be able to get up-to-date defect metrics for analysis.

Defect Report
After uncovering a defect (bug), testers generate a formal defect report. The purpose of a defect report is to state the problem as clearly as possible so that developers can replicate the defect easily and fix it. DEFECT REPORT TEMPLATE In most companies, a defect reporting tool is used and the elements of a report can vary. However, in general, a defect report can consist of the following elements. ID Project Product Release Version Module Unique identifier given to the defect. (Usually Automated) Project name. Product name. Release version of the product. (e.g. 1.2.3) Specific module of the product where the defect was detected.

Detected Build Version Summary Description

Build version of the product where the defect was detected (e.g. 1.2.3.5) Summary of the defect. Keep this clear and concise. Detailed description of the defect. Describe as much as possible but without repeating anything or using complex words. Keep it simple but comprehensive. Step by step description of the way to reproduce the defect. Number the steps. The actual result you received when you followed the steps. The expected results. Attach any additional information like screenshots and logs. Any additional comments on the defect. Severity of the Defect. (See Defect Severity) Priority of the Defect. (See Defect Priority) The name of the person who reported the defect. The name of the person that is assigned to analyze/fix the defect. The status of the defect. (See Defect Life Cycle) Build version of the product where the defect was fixed (e.g. 1.2.3.9)

Steps to Replicate Actual Result Expected Results Attachments Remarks Defect Severity Defect Priority Reported By Assigned To Status Fixed Build Version

REPORTING DEFECTS EFFECTIVELY

It is essential that you report defects effectively so that time and effort is not unnecessarily wasted in trying to understand and reproduce the defect. Here are some guidelines:

Be specific: o Specify the exact action: Do not say something like Select ButtonB. Do you mean Click ButtonB or Press ALT+B or Focus on ButtonB and click ENTER? Of course, if the defect can be arrived at by using all the three ways, its okay to use a generic term as Select but bear in mind that you might just get the fix for the Click ButtonB scenario. [Note: This might be a highly unlikely example but it is hoped that the message is clear.] o In case of multiple paths, mention the exact path you followed: Do not say something like If you do A and X or B and Y or C and Z, you get D. Understanding all the paths at once will be difficult. Instead, say Do A and X and you get D. You can, of course, mention elsewhere in the report that D can also be got if you do B and Y or C and Z. o Do not use vague pronouns: Do not say something like In ApplicationA, open X, Y, and Z, and then close it. What does the it stand for? Z or, Y, or X or ApplicationA? Be detailed: o Provide more information (not less). In other words, do not be lazy. Developers may or may not use all the information you provide but they sure do not want to beg you for any information you have missed. Be objective: o Do not make subjective statements like This is a lousy application or You fixed it real bad. o Stick to the facts and avoid the emotions. Reproduce the defect: o Do not be impatient and file a defect report as soon as you uncover a defect. Replicate it at least once more to be sure. (If you cannot replicate it again, try recalling the exact test condition and keep trying. However, if you cannot replicate it again after many trials, finally submit the report for further investigation, stating that you are unable to reproduce the defect anymore and providing any evidence of the defect if you had gathered. ) Review the report: o Do not hit Submit as soon as you write the report. Review it at least once. Remove any typos

Defect Density
DEFINITION

Defect Density is the number of confirmed defects detected in software/component during a defined period of development/operation divided by the size of the software/component. ELABORATION The defects are:

confirmed and agreed upon (not just reported). Dropped defects are not counted.

The period might be for one of the following:

for a duration (say, the first month, the quarter, or the year). for each phase of the software life cycle. for the whole of the software life cycle.

The size is measured in one of the following:

Function Points (FP) Source Lines of Code

DEFECT DENSITY FORMULA

USES

For comparing the relative number of defects in various software components so that high-risk components can be identified and resources focused towards them. For comparing software/products so that quality of each software/product can be quantified and resources focused towards those with low quality.

Smoke Testing
DEFINITION Smoke Testing, also known as Build Verification Testing, is a type of software testing that comprises of a non-exhaustive set of tests that aim at ensuring that the most

important functions work. The results of this testing is used to decide if a build is stable enough to proceed with further testing. The term smoke testing, it is said, came to software testing from a similar type of hardware testing, in which the device passed the test if it did not catch fire (or smoked) the first time it was turned on.

ELABORATION Smoke testing covers most of the major functions of the software but none of them in depth. The result of this test is used to decide whether to proceed with further testing. If the smoke test passes, go ahead with further testing. If it fails, halt further tests and ask for a new build with the required fixes. If an application is badly broken, detailed testing might be a waste of time and effort. Smoke test helps in exposing integration and major problems early in the cycle. It can be conducted on both newly created software and enhanced software. Smoke test is performed manually or with the help of automation tools/scripts. If builds are prepared frequently, it is best to automate smoke testing. As and when an application becomes mature, with addition of more functionalities etc, the smoke test needs to be made more expansive. Sometimes, it takes just one incorrect character in the code to render an entire application useless. ADVANTAGES

It exposes integration issues. It uncovers problems early. It provides some level of confidence that changes to the software have not adversely affected major areas (the areas covered by smoke testing, of course)

LEVELS APPLICABLE TO

Smoke testing is normally used in Integration Testing, System Testing and Acceptance Testing levels. NOTE Do not consider smoke testing to be a substitute of functional/regression testing.

Regression Testing
DEFINITION Regression testing is a type of software testing that intends to ensure that changes (enhancements or defect fixes) to the software have not adversely affected it. ELABORATION The likelihood of any code change impacting functionalities that are not directly associated with the code is always there and it is essential that regression testing is conducted to make sure that fixing one thing has not broken another thing. During regression testing, new test cases are not created but previously created test cases are re-executed. LEVELS APPLICABLE TO Regression testing can be performed during any level of testing (Unit, Integration, System, or Acceptance) but it is mostly relevant during System Testing. EXTENT In an ideal case, a full regression test is desirable but oftentimes there are time/resource constraints. In such cases, it is essential to do an impact analysis of the changes to identify areas of the software that have the highest probability of being affected by the change and that have the highest impact to users in case of malfunction and focus testing around those areas. Due to the scale and importance of regression testing, more and more companies and projects are adopting regression test automation tools. LITERAL MEANING OF REGRESSION Regression [noun]: the act of going back to a previous place or state; return or reversion.

ANALOGY You can consider regression testing as similar to moonwalk or backslide, a dance technique (popularized by Michael Jackson) that gives the illusion of the dancer being pulled backwards while attempting to walk forward. Well, many will not agree with this analogy but what the heck! Lets have some fun!