Performance Test Strategy For XXXXXXX Phase 1a

DRAFT

TABLE OF CONTENTS
REVISION HISTORY................................................................3 1 2 3 INTRODUCTION................................................................5 TEST OBJECTIVES..............................................................5 TEST SCOPE ....................................................................7 3.1 PERFORMANCE TEST SCOPE ...................................................................7 Goals of Performance Tests.............................................................7 3.2 PERFORMANCE TESTING SCENARIOS..........................................................8 ELEMENTS OF WORK IN PERFORMANCE TEST SCOPE.............................................10 3.4 ELEMENTS OF WORK NOT IN PERFORMANCE TEST SCOPE................................10 4. TEST APPROACH............................................................10 PERFORMANCE TEST APPROACH.....................................................................11 4.1.1 Stage 1: Test Strategy Development.................................11 4.1.2 Stage 2: Requirements Review..........................................11 4.1.3 Stage 3: Test Script Design and Development...................11 4.1.4 Stage 4: Test Script Execution...........................................11 5. TEST DELIVERABLES......................................................12 5.1 PROJECT TEST DELIVERABLES................................................................12 5.2 PERFORMANCE TEST DELIVERABLES.........................................................12 6. TEST ENVIRONMENT REQUIREMENTS..............................12 CONFIGURATION.......................................................................................12 TEST DATA............................................................................................13 7. ASSUMPTIONS, DEPENDENCIES, RISKS............................13 7.1 ASSUMPTIONS..................................................................................13 7.2 DEPENDENCIES................................................................................13 7.3 RISKS..........................................................................................14 8. PERFORMANCE TEST WORK PLAN...................................14

Revision History
Each time this document is modified, document the date, the persons name making the change and a brief description of the change. Date
9/10/2001 11/13/200 1

Author

Description of Change
Draft Modified Script Names & Scenarios and Test Goals.

INTRODUCTION

This document details the performance test strategy of the XXXXX system project. This document includes information about the tests objectives, scope, and approach. In addition, it provides a list of the roles and responsibilities for personnel participating on the test, as well as operational procedures that will be used during the test and a description of the test environment requirements. The document sections are organized as follows: Test Objectives describes the overall test objectives. Test Scope describes the overall approach to testing. The approach provides a list of testing deliverables. Test Approach identifies the methodology that will be used in testing the product. Test Deliverables outline the deliverables of Phase 1a.

Test Environment Requirements describes the hardware, software, data, and staffing needs for the test effort. This section also describes any dependencies on the development environment. Assumptions, Dependencies, and Risks describes the assumptions, dependencies, and risks of the projects performance tests. Performance Test Work Plan describes the main milestones and outlines the project plan for testing of the product.

TEST OBJECTIVES

The objectives of the Perf. Test Team are to ensure performance test scripts are generated and executed, and the corresponding test execution reports are created for the XXXXX performance test project. To determine the scope of the testing requirements, the Performance Test Engineer conducted an assessment of the XXXXX system technical specifications. After the requirements have been identified, summary and detailed test scripts will be constructed using the Mercury Interactive LoadRunner testing software. Estimates for the Test Script creation will be based on the creation of one test script per business process, however, several different permutations may require testing and this validation requirement would necessitate increasing the number of test scripts and the test script preparation estimated time. The main objective of performance testing is to help ensure the XXXXX system performs within set performance guidelines for the selected functional transactions running under varying user load scenarios. The performance test team will work with the development team to define the required functional transactions, user load scenarios, and performance guidelines.

TEST SCOPE Performance Test Scope

3.1

The performance test effort will focus on developing and executing a performance test model for the XXXXX system. A performance test model consists primarily of virtual user scripts and load scenarios. All activities required to perform a variety of XXXXX system transactions will be recorded into a virtual user script. The virtual user script is used to simulate one or more users executing the business process. A load scenario defines what virtual user scripts will be executed, how many simulated users will execute each virtual user script, and how many iterations of each virtual user script is scheduled to be executed by each user. We will assist the development team in determining how many concurrent will be used and the amount/duration of think time users that will be included in the performance test. We will also assist in determining the transaction mix, as well as the speed setting for LoadRunner, which can handle modem speeds from 2400 baud up to T1 speed. Goals of Performance Tests 1. Meet or exceed the following volume impact on the network based on the following figures: Yearly Annual
2005

Expected No. of DIPs per year / per day / per hour

20,563 / 66 / 9

Expected No. of Completed Applications per year / per day / per hour
17,136 / 55 / 7

2. Meet or exceed the following performance target figures from a user satisfaction perspective: Business Function
Content Pages ( Product, Brochure & Company Info) Registration Logon/Logoff Calculators Decision in Principle Application Form Quick Address Query

Response Time
95% within 3 seconds from screen to screen 95% within 5 seconds from request to authorization 95% within 5 seconds from request to response 95% within 3 seconds from screen to screen using Save or Back buttons 95% within 3 seconds from screen to screen using Save or Back buttons 95% within 3 seconds from screen to screen using Save or Back buttons 95% within 3 seconds from screen to screen using Save or Back buttons

Calculator
DIP request Case Tracking Application Transfer

95% within 15 seconds from request to response

95% within 30 seconds from request to response 95% within 15 seconds from request to response 95% within 30 seconds from request to response

3.2

Performance Testing Scenarios

Scenario 1 PropTestCombo25 Group Name

PropTestCombo1 PropTestCombo1_1 PropTestCombo1_2 PropTestCombo1_3 PropTestCombo1_4 PropTestCombo1_5 PropTestCombo1_6 PropTestCombo2 PropTestCombo2_1 PropTestCombo2_2 PropTestCombo3 PropTestCombo4 PropTestCombo4_1 PropTestCombo4_2 PropTestCombo4_3 PropTestCombo4_4 PropTestCombo4_5 PropTestCombo5 PropTestCombo6 PropTestCombo6_1 PropTestCombo6_2 PropTestCombo6_3 PropTestCombo6_4 PropTestCombo6_5 PropTestCombo7 PropTestCombo8

Script Name
PropTestCombo1 PropTestCombo1 PropTestCombo1 PropTestCombo1 PropTestCombo1 PropTestCombo1 PropTestCombo1 PropTestCombo2 PropTestCombo2 PropTestCombo2 PropTestCombo3 PropTestCombo4 PropTestCombo4 PropTestCombo4 PropTestCombo4 PropTestCombo4 PropTestCombo4 PropTestCombo5 PropTestCombo6 PropTestCombo6 PropTestCombo6 PropTestCombo6 PropTestCombo6 PropTestCombo6 PropTestCombo7 PropTestCombo8

Vuser Quantity
10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10

Total number of vusers = 250

Scenario 2 PropTestCombo50 Group Name

PropTestCombo1 PropTestCombo1_1

Script Name
PropTestCombo1 PropTestCombo1

Vuser Quantity
10 10

PropTestCombo1_2 PropTestCombo1_3 PropTestCombo1_4 PropTestCombo1_5 PropTestCombo1_6 PropTestCombo2 PropTestCombo2_1 PropTestCombo2_2 PropTestCombo3 PropTestCombo3_1 PropTestCombo4 PropTestCombo4_1 PropTestCombo4_2 PropTestCombo4_3 PropTestCombo4_4 PropTestCombo4_5 PropTestCombo4_6 PropTestCombo4_7 PropTestCombo5 PropTestCombo6 PropTestCombo6_1 PropTestCombo6_2 PropTestCombo6_3 PropTestCombo6_4 PropTestCombo6_5 PropTestCombo6_6 PropTestCombo6_7 PropTestCombo7 PropTestCombo8 PropTestCombo8_1 PropTestCombo8_2 PropTestCombo8_3

PropTestCombo1 PropTestCombo1 PropTestCombo1 PropTestCombo1 PropTestCombo1 PropTestCombo2 PropTestCombo2 PropTestCombo2 PropTestCombo3 PropTestCombo3 PropTestCombo4 PropTestCombo4 PropTestCombo4 PropTestCombo4 PropTestCombo4 PropTestCombo4 PropTestCombo4 PropTestCombo4 PropTestCombo5 PropTestCombo6 PropTestCombo6 PropTestCombo6 PropTestCombo6 PropTestCombo6 PropTestCombo6 PropTestCombo6 PropTestCombo6 PropTestCombo7 PropTestCombo8 PropTestCombo8 PropTestCombo8 PropTestCombo8

10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 50 50 10 10 10 10 10 10 10 50 50 10 10 10 10 10

Total number of vusers = 500

During the test execution stage of the performance test, each planned load scenario will be executed. The system performance of the XXXXX system will be monitored and tracked during execution of each load scenario. For each test scenario, the following conditions apply: All tests will ramp up 1 vuser every 15 seconds.

 Each test group will be selected to start randomly. Think times of 20 seconds per page are utilized. All tests will utilize LAN speed (100mps). All tests will run for an hour unless otherwise specified.

Elements of Work in Performance Test Scope

The following provides a list of the high-level activities to be considered in scope for the performance test: Identification of the performance requirements. Creation of the automated virtual user scripts. Creation of the automated load scenarios plus the transaction mix. Setting up the test environment for the test. Setting up the test data required for the test. Execution of the performance test model. Enter performance defects in Excel spreadsheet. Reporting test execution results/status (i.e. daily/weekly meetings, defect reporting).

3.4

Elements of Work not in Performance Test Scope

The following provides a list of the high-level activities to be considered out of scope for the testing team: Fixing defects. Unit testing the software components of the XXXXX system.

4.

TEST APPROACH

The approach to performance testing consists of executing testing in stages, which become incorporated into the overall project development life cycle. Multiple execution cycles may occur in the course of the testing project, based on the assessment of the quality of the test.

The generic high-level stages, which will be performed by the test team during the course of testing the XXXXX system, will include: Stage 1 Test Strategy Development. Creation of the XXXXX Performance Test Strategy and high-level testing project work plan. Stage 2 Requirements Review. Development of the XXXXX Performance Test Plan. Stage 3 Test Script Design and Development. Preparation of test data, test cycle planning and development of detailed test scripts. Stage 4 Test Script Execution. Execution of test scripts, management of test results and utilization of defect tracking process.

Performance Test Approach

4.1.1 Stage 1: Test Strategy Development This document represents the Test Strategy. 4.1.2 Stage 2: Requirements Review The performance requirements for the project will be determined by reviewing the High Level Design document and the Detailed Design document, reviewing the Project Requirements document and, when needed, conducting interviews with the subject matter experts for the application At the conclusion of this stage, transaction mixes and load scenarios should be defined for the performance test. The automated performance-testing tool uses the Perfmon and Netmon utilities of Windows/NT to gather execution statistics from the performance test. The Perf. test group should identify the performance statistics to be gathered as a result of the performance test. 4.1.3 Stage 3: Test Script Design and Development During this stage of the performance test, the automated load scenarios will be built with the LoadRunner Virtual User Generator and the LoadRunner Controller. Load scenario development includes recording the selected transactions into virtual user scripts, parameterize input data for the scripts, building any required verification points in the scripts, creating/configuring user groups, and assigning virtual user scripts to user groups. During this stage, the Perf. test group should configure the server monitor to capture the performance statistics they require for the web server, the application server, and the database server. 4.1.4 Stage 4: Test Script Execution During the test execution stage, the planned load scenarios are executed via the LoadRunner Controller and Scheduler. Performance problems encountered during test execution will be entered into an Excel spreadsheet as performance defects. After the defect fix is complete, the defect fix will be migrated into the test environment and regression tested by the test team. If the defect fix requires a change to the products application code, it should be functionally tested first to ensure that it does

not affect the functionality of the product. After the performance functional regression test is complete, the load scenario should be re-executed to determine the defect fixs impact on performance. If the defect fix requires a change to the technical environment (e.g., web server configuration, application server configuration, database server, etc), the load scenario should be reexecuted to determine the defect fixs impact on performance.

5.

TEST DELIVERABLES

The Performance Test deliverables (indicated with a check mark) are:

5.1

Project Test Deliverables

Performance Test Strategy (this document) Status Reports Performance Test Summary Document

5.2

Performance Test Deliverables

Virtual User Scripts (LoadRunner) Load Scenarios (LoadRunner) Test Execution Reports (LoadRunner)

6.

TEST ENVIRONMENT REQUIREMENTS

Configuration

COMPONENT
Test Client Workstations Web Servers

HARDWARE
Pentium III 500mhz 384mb RAM Compaq DL 380 rack mounted unit Dual Pentium III 1ghz Xeon CPU 1GB RAM 18.2gb SCSI mirrored drive Compaq DL 380 rack mounted unit Dual Pentium III 1ghz Xeon CPU 1GB RAM 18.2gb SCSI mirrored drive Compaq DL 580 rack mounted unit Dual Pentium III 1ghz Xeon CPU 1GB RAM 18.0gb RAID 5 drive 54gb total

SOFTWARE
Windows NT 4.0 Workstation Explorer 5.01 or higher Windows NT Server 4.0 SP6a IIS 4.0 SQL Client SMTP Windows NT Server 4.0 SP6a XXXXX Windows NT Server 4.0 SP6a SQL Server 7.0 SP2

Application Servers

Database Servers

Test Data
Test scripts will be written to encompass the most complex portion of the identified business process utilizing complex data, and if one test script does not accomplish the entire test for the business process, multiple test scripts will be created.

7. 7.1

ASSUMPTIONS, DEPENDENCIES, RISKS Assumptions

The test lab is complete; all web servers, application servers, and database servers have been acquired, installed, and tested. The test lab includes client desktops running all of the browser/operating system combinations required to test the XXXXX system. The test lab provides access to third party vendors. The test labs databases have sufficient data to test all major functional business requirements of the XXXXX system. All software components of the XXXXX system have been unit and integration tested by the development team. All requirements for the project have been implemented into the product.

7.2

Dependencies

 The test environment must be functional. The XXXXX application software must function properly and be stable in order to complete Load/Performance testing.

7.3

Risks

If any of the assumptions or dependencies is not met, the XXXXX performance test project may not be completed on time or may go over budget.

8.

PERFORMANCE TEST WORK PLAN

Preparation Develop Test Plan .. 8/219/14/01 Configure Test Servers 8/27-9/14/01 Database Load . 9/10-9/14/01 Software Installation Final Code Release ... 9/11/01 Final Installation Instructions Release .. 9/11/01 Software Installation .... 9/11-9/14/01 Validate Installation .. 9/14/01 Test Execution Develop Test Scripts 9/179/19/01 Cycle 1 Tests Begins 9/199/26/01 PROP Test 1 Script .. 9/19/01 PROP Test 2 Script .. 9/20/01 PROP Test 3 Script .. 9/21/01 PROP Test 4 Script .. 9/24/01 Retests... 9/25/01 Retests .. 9/26/01 Cycle 2 Code Release .. 9/27/01 Cycle 2 Installation Instructions Release . 9/27/01 Cycle 2 Software Installation 9/27-10/02/01 Cycle 2 Validate Installation 10/02/01 Cycle 2 Tests Begins . 10/0310/10/01 PROP Test 1 Script ... 10/03/01 PROP Test 2 Script ... 10/04/01 PROP Test 3 Script ... 10/05/01 PROP Test 4 Script . 10/08/01 Retests . 10/09/01

Retests . . 10/10/01 Production Implementation Installation Certification 10/11/01 Production Installation .. 10/12/01