End To End Performance Process

Software Test & Performance
September 24-26, 2008
508 End-to-End Performance Process:

From Requirements to Production Monitoring
Alfred Wong
2
About me…
• 5+ years of experience in performance testing /
engineering area
• EE  SDET  PTE  PTM  PTA 
• Performance Test Advisor role:

– Provide technical consultation to new projects on performance
aspects.
– Advise performance certification and testing of web layer,
middle-tier and backend systems.
– Responsible for performance test team improvements on
standards and quality.
– Drive cross team performance tuning and capacity planning
efforts, as well as overall performance improvements across the
company.
508 End-to-end Performance Process | © 2008 by Alfred Wong; made available under the EPL v1.0
3
About you…
• Test Engineer / QA Analyst
• Performance Analyst
• Performance Test Engineer
• Performance Test Lead / Manager
• QA or Test Lead / Manager
• QA or Test Director
• Performance team structure?

– Centralized / Service based
– Integrated
4
Agenda
• Scope
• Typical Performance Testing
• End to End Performance Process
• Value-add for EEPP
• Putting the pieces together
• Challenges
• Q&A + Comments
5
Scope
• Let’s build an online bookstore!

– User accounts & profile
– Search (by title, author, keyword, ISBN etc)
– Availability & Prices
– Check-out
6
Scope
• N-tier ecommerce architecture & all modules /
components / services associated
Services E-mail
Content
App Server Database Management etc

Web
Payment Processing
Media Storage
• Yet, principles are applicable to most software projects
7
Software Development Life Cycle

Functional Overall
Integration Project
Dev / B&D / Test / Perf
Regression Sign-off
often occur in parallel to
etc..
varying degrees as needed
Business
Requirements
Production
Project Development
B&D General Performance Go-Live / Monitoring &
Specification & Unit Testing Test Test & Tune Launch Maintenance
Functional Build & Functionality

Dev / B&D / Test /
Architecture Deployment Stability
Perf is typically an
Dev design in Test Performance
iterative process
Test design Environment Capacity
Operation
8
At which stage is the

performance team engaged?
9
Typical Performance Testing Focus

Production
Project Development General Performance Go-Live / Monitoring &
B&D
• Performance test planning

• Performance test execution
– Test scripts
– Load generation
– Monitoring system under test
• Perf Analysis & Tuning
• “Can we launch?”
10
I am assuming that you…

• understand the overall system architecture
• have performance testing goals / objectives /
requirements
• are familiar with test & production environments,
as well as their differences
• can model user behavior / work load pattern
• are able to use test tools correctly
• are experienced in interpreting test data (test
tool & server metrics)
11
End to End Perf Process Principles

Production
B&D
• Think beyond typical performance testing

• Look at the entire process end to end
– Business Requirement
– Project Specification: functional / architecture /
development / test / operation
– Development & Unit Testing
– Build & Deployment
– Performance Test / Analysis / Tuning
– Production Monitoring & Maintenance
12
Can the Performance Team

add value throughout?
13
Business Requirements
• Typical:
– Summarize background & current situation
– Outline business objectives, primary customers & scenarios /
business use cases
– Describe high level requirements & specifies all basic features
– Define scope / limits & success criteria / metrics / measurement
– Cost Benefit Data / ROI analysis
– Schedule & budget
• EEPP:
– Performance requirements (what's considered 'good enough'?)
– Traffic forecast information / ROI: how does project impact traffic
volume / pattern?
– Schedule & budget
• Performance test schedule
• Hardware budget for lab & production
14
Functional Specification
Production
B&D
• Typical:
– outline the capabilities & behavior of product
– describe user interaction
– list properties of inputs & outputs
– define the logic of user experience
– illustrate product appearance (screenshots)
• EEPP
– outline monitoring & measurement needs
– define overall performance metrics & targets
15
Architecture Spec
• Typical
– general system architecture & strategies
– sub-system architecture
• definition
• constraints
• responsibilities
• resources
• processing
• interface
• inter-dependencies etc
• EEPP
– Performance test strategies
– Performance monitoring
– where & how?
16
Dev Design Spec

• EEPP
– Performance metrics & targets
– Performance monitoring
– Performance testability
– Unit Performance Test
• Memory management
• Thread Pool
• Connections (to DB and other app servers)
• Queuing & Queues
17
Performance Test Spec

• Entrance Criteria
• Dependencies & SLA
• Test environment
• Test cases & scenarios (workload)
• Ratio of different workloads
• Performance Tests
• Performance goals & targets
• Performance analysis
• Sign-off Criteria
18
Operation Specification
• EEPP
– hardware platform
– system level performance metrics & targets
– performance monitoring systems integration
– threshold for alerts
– performance issue mitigation steps
– escalation paths
– capacity management integration
19
Development
Production
B&D
• EEPP:
– Performance comparison of different
algorithms
– Prototype performance validation
– Unit performance testing
– Performance / code profiling
– Integrate unit performance tests & build
process
20
Performance testing & tuning

Production
B&D
• Typical:
– Given code / system to test
– Focus on:
• 'testing'
• load Generation
• system behavior analysis
• EEPP:
– Traffic profile / pattern analysis
– Validate performance monitoring mechanism
– Correlate unit performance testing results
– Provide performance tuning recommendations
– Performance bugs investigation (session 608)
21
After Launch
Production
B&D
• EEPP:
– Production performance monitoring
– Correlate performance test results
– Validate performance change
– Performance issues investigation
– Capacity planning & management
22
Value-add for EEPP

• Project design phase
– gradual knowledge transfer
– integrated team
– realistic schedule
– built-in performance considerations upfront
• Project implementation phase
– performance info throughout life of project
– bug prevention vs bug discovery
– simplify performance investigation
– relevant performance results
• Higher quality / confidence / predictability

23
Putting the pieces together

• How to implement?
– VISION
• Goals / Justification / Roadmap / Buy-off
– necessity is the mother of invention
• Adapt principles to specific situations
– KNOWLEDGE
• of overall architecture
• of architecture & dev design
• of project team on what you (the performance expert) can do!
– EXECUTION
• think big, start small
• best practices are learnt through experience
24
So, how do you know if you have

succeeded?
• SMART goals
– Specific, Measurable, Attainable, Realistic, Timely / Tangible
• Quotes of what others say about the performance test

team (before & after):
– Management / Architect / Developer / Tester / PM / Operations
• How test results add value:

– end user experience
– stability / scalability / capacity info
– hardware platform
– tuning parameters
– predict system level performance in production
– efficient production performance issue investigation
25
Challenges? What challenges?

• Push-backs / Fear
• Bug count
• Gaps:
– budget
– process
– people / skill set
– technology / tools
– time
– anything else?
26
Q&A + Comments
as a performance engineer!

End To End Performance Process

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

End To End Performance Process

Hochgeladen von

Copyright:

Verfügbare Formate

Software Test & Performance

September 24-26, 2008

508 End-to-End Performance Process:

• Performance Test Advisor role:

• Performance team structure?

• Let’s build an online bookstore!

App Server Database Management etc

Software Development Life Cycle

Functional Build & Functionality

At which stage is the

Typical Performance Testing Focus

• Performance test planning

I am assuming that you…

End to End Perf Process Principles

• Think beyond typical performance testing

Can the Performance Team

Dev Design Spec

Performance Test Spec

Performance testing & tuning

Value-add for EEPP

• Higher quality / confidence / predictability

Putting the pieces together

So, how do you know if you have

• Quotes of what others say about the performance test

• How test results add value:

Challenges? What challenges?

Das könnte Ihnen auch gefallen