4/17/2009

Best Practices in Performance Engineering

Prashant Gautam Wipro Technologies

2008 Wipro Ltd - Confidential

Performance Engineering?
What? Performance Engineering is a set of activities for quantifying, measuring and analyzing in each phase of Systems Development Lifecycle whether the solution being implemented meets the non functional/ performance requirements agreed for the solution. Why? Better Project Governance Defined Standards & Methodologies Early estimation of Performance Risks Optimum Resource Utilization Lowering Cost of Ownership Knowledge Management

2008 Wipro Ltd - Confidential

Confidential 2008 Wipro Ltd

4/17/2009

Performance Engineering v/s Performance Testing

2008 Wipro Ltd - Confidential

Performance Engineering Approach

2008 Wipro Ltd - Confidential

Confidential 2008 Wipro Ltd

4/17/2009

Performance Engineering Life Cycle

Requirements
Approved Performance requirement document

R R
Predict
a Approved Performance Assurance Test Plan

Plan / Design

Modeling

Performance Assurance Lifecycle

Develop Scripts

Validation of scripts and data.

R
Tuning
R Indicates Review Process

Execute /Monitor

Smoke, Baseline and all planned test are executed and monitored.

2008 Wipro Ltd - Confidential

Performance Centric Development Methodology

SDLC Lifecycle

Requirement Phase

Architecture & Design Phase

Production / Construct Test Phase Maintenance Phase

Performance Requirement Elicitation Performance Requirement Validation Workload Modeling & Analysis

Architecture Performance Assessment Infrastructure Sizing & Capacity Planning Test Strategy / Plan Development

Component performance testing Stub / Driver based integrated components Performance testing Test Data Management

Load Test Stress Test

Application Performance Management

Deployment

Volume Test Endurance Test Fail over / Back Test Application / System Scalability Assessment Capacity Modeling & Performance Forecast

Trend Analysis on workload, utilization etc. & Impact Assessment Capacity Planning , Capacity Upgrade Forecast & Roadmap

Confidential Copyright 2007 Wipro Ltd 2008 Wipro Ltd - Confidential

Confidential 2008 Wipro Ltd

4/17/2009

Performance Engineering - Overview

PRE PRODUCTION PHASE
Requirement Management Design Development System & Functional Test Testing UAT

POST PRODUCTION PHASE

Production

PERFORMANCE ENGINEERING

PERFORMANCE MONITORING

Performance Requirements & Benchmarks Test Repositories ( Performance Test Strategy, Scripts, Test Data ) Test Execution & Analysis Reports Test Environment Management Knowledge Repository

APPLICATION MAPPING & SLAs Test Repositories ( Monitoring Strategy, Scripts ) Monitoring & Analysis Reports Test Tool Config & Management Knowledge Repository

Shared Resource Pool & Tool Infrastructure Frameworks, Standards & Best Practices

2008 Wipro Ltd - Confidential

Performance Engineering Pre Production Activity

Evaluate System & Understand Requirement

Develop test assets

Benchmarks Existing

No

Profiling and benchmarking

Criteria not met

Execute tests against benchmarks

Tune

Analyze results

Identify exploratory tests

Test Summary & Sign off

2008 Wipro Ltd - Confidential

Confidential 2008 Wipro Ltd

4/17/2009

Performance Engineering Post Deployment Activity

Define Key Objectives & SLAs Understand Business Impact

Monitor the Profiles, Appn, Infrastructure

System In Good Health

YES Results Match SLA?

NO

Raise Alert

Implement Changes In Production

Tuning/Redo Capacity Plann

Application Triage & Diagnostics

2008 Wipro Ltd - Confidential

Performance Engineering Best Practices

Requirement Gathering Planning Design Execution Analysis Reporting

2008 Wipro Ltd - Confidential

Confidential 2008 Wipro Ltd

4/17/2009

Best Practices Requirement Gathering

Requirements Walkthrough session: Conduct requirement walkthrough / review session during the requirement definition / change phases with core requirements team Centralized requirement repository: Use RTM( Requirement Traceability Matrix) to manage the requirement. It helps in forward and backward traceability RTM Tools Req Pro, QC Create Requirement Stability Matrix Core requirements team: Creation of core requirements management team that includes representation for all the project team for review and analyze impact of change like Business Analyst, Business User, SMEs Availability and usage of requirement review checklist / Guidelines: Requirements team should use and mark identified activities completed to ensure consistency and completeness. Sign off of the Requirement Document
2008 Wipro Ltd - Confidential

Best Practices Planning

Use Estimation Model: Using a structured and standardized estimation process with tool, provides a reliable prediction of effort for the project. Compare and Contrast: Compare and Contrast the values generated by the estimation tool and actual effort put on accomplishing the tasks across phases. Refer Historical Data: Refer to historical data to understand the deviations and reasons for the same to resolve estimation gaps. Normalization of Complexities: Normalization of complexity across projects will ensure easy references across similar projects to arrive at accurate estimates. Document Assumptions and Dependencies: Clearly document assumptions and dependencies. And the same should be revisited during the requirements and design phase for their applicability and resolution

2008 Wipro Ltd - Confidential

Confidential 2008 Wipro Ltd

4/17/2009

Best Practices Design

Follow Scripting Standards: It provides reusability, ease of review and consistency across the scripts.
Scripting Standards

2008 Wipro Ltd - Confidential

Best Practices Design (Contd)

Performance Modeling: The process of PM involves accomplishment of following steps. Identify Key Scenarios: Identify scenarios where performance is important and scenarios that pose the most risk to performance objectives Identify Workload: Identify how many users and how many concurrent users system needs to support Identify Performance Objectives: Identify performance objectives for each of the key scenarios. Performance objectives are business requirements. Identify Budget: Identify budget constraints. This includes maximum execution time in which the operation needs to be completed, resource utilization constraints.

1.

2. 3.

4.

2008 Wipro Ltd - Confidential

Confidential 2008 Wipro Ltd

4/17/2009

Best Practices Design (Contd)

5. 6. Identify processing steps: Break down scenarios into component processing steps Allocate Budgets: Spread the budget, determined in step 4, across processing steps, determined in step 5, to meet performance objective determined in step 3. Evaluate: Evaluate design against objectives and budgets. Design might need to undergo changes or need to spreading of response time and resource utilization budget differently will be required to meet the performance objectives

7.

2008 Wipro Ltd - Confidential

Best Practices Execution

Baseline and Benchmark - Baselines and benchmarks are generally the first and last tests you will execute, respectively Baseline is the process of running a set of tests to capture performance metric data for the purpose of evaluating the effectiveness of subsequent performanceimproving changes to the system or application. Benchmarking is the process of comparing your system performance against an industry standard that is endorsed by some other organization.

2008 Wipro Ltd - Confidential

Confidential 2008 Wipro Ltd

4/17/2009

Best Practices Execution (Contd)

Monitoring is collection of server, network,application performance metrics and time series graphs by using Tools

2008 Wipro Ltd - Confidential

Best Practices Analysis

Application Performance metrics System Resource Utilization metrics

Analysis

Application Performance

System Resource Utilization

2008 Wipro Ltd - Confidential

Confidential 2008 Wipro Ltd

4/17/2009

Best Practices Reporting

Report is the most important tangible deliverables that is produced during the engagement. Reporting is the process of presenting results data that will support key technological and business decisions. The most effective report will present analysis, comparisons, and details of the execution results, with observation & recommendations that will have an influence on critical business decision-making. A comprehensive final report should be structured in the following manner (but not limited to). Executive Summary Holds the crux of the engagement in a summarized form. Re-cap Acceptance Criteria (Requirements) Re-Cap Workloads Actually Used Summarize Tests Individually Summarize Tuning Activity (System Modifications) Summarize Results Collectively Document Conclusions & Recommendation
2008 Wipro Ltd - Confidential

Case Study
Client Profile : Global Leader in the Oil and Natural Gas Industry. Engagement Scope
Central Finance organization Functionalities Finance, Timewriting and Cost Center Allocation for employees and contractors Geographies USA, UK, Netherlands, Malaysia, Poland. Project would retire several Legacy Systems.

Testing Snapshot
Definition of a Performance Test Strategy. Planning & execution of Load, stress, volume and endurance tests Application Profiling & Tuning Assistance

Size, Application & Tools

Duration/Team
Duration: 9 months Team Size: Avg. 3, Peak 5

Environment:
SAP ECC 5.0 FI/SD/MM/PS/BW/Portals

Tools
HP LoadRunner HP SiteScope Wipro eProfiler

Pain Areas
No prior experience in onsite offshore delivery model Aggressive project plan with multiple project phases / testing activities in parallel Lack of performance testing skills in-house. Coordination amongst distributed Business Process Owners & Support Groups to ensure comprehensive testing. Team distributed across US, UK, Netherlands and India

How Wipro helped

Defined a Performance Test Strategy for the project. Deployed an Offshore-centric Test Organization to optimize time & cost, with a very small Onsite Team [15%]. Defined performance benchmarks for a user base of 1000+ concurrent users Nearly 80 performance scripts were developed and executed using HP Load Runner Tool. Identified application and database level performance issues using Wipro eProfiler tool. Assisted Development team during performance tuning activities.

Key Challenges
Clarifying the role of IV&V and setting expectations. Involving Business Process Owners in Knowledge Transfer, providing clarifications and reviewing of performance test scenarios and test data for test execution. Data preparation in SAP for test execution in different test phases. Coordination with geographically distributed Teams for test environment setup, test tool setup & administration, test data, etc.

2008 Wipro Ltd - Confidential Confidential Copyright 2008 Wipro Ltd

Confidential 2008 Wipro Ltd

10

4/17/2009

Performance Engineering Solutions

Performance Engineering Solution tools are provided by the following vendors:

2008 Wipro Ltd - Confidential

Thank You
Prashant Gautam
gautam.prashant@wipro.com

7th April 2009

2008 Wipro Ltd - Confidential 2009

22

Confidential 2008 Wipro Ltd

11