Tech Talk: Performance benchmark

Tech Talk: Performance benchmark

 Number of Number of times Number of times Avg. Test Time
Maximum Avg. Test Time (sec) Avg. Test Time (sec)
Scenario times executed executed executed (sec)
Scenario acceptable target
ID (throughput for (throughput for (throughput for
from business (iteration 1) (iteration 2) (iteration 3)
iteration 1) iteration 2) iteration 3)
994 executions 1470 executions 194 seconds = ~3 minutes 1 minute 52 seconds
From iteration 2: 541 seconds = ~9
#11 Sales order (flow A-2) 342 executions From iteration 1: +190% 3 minutes From iteration 1: -64% From iteration 2: -42%

Key benchmark deliverables

+47% minutes
Total gain: +329% Total gain: -79%

➢ Performance benchmark report

1 minute 44
951 executions 1471 executions 187 seconds = ~3 minutes
seconds

➢ Issues detected/fixed in each

#8 Sales order normal flow 337 executions From iteration 2: 3 minutes 534 = ~9 minutes
From iteration 1: +182% From iteration 1: -65% From iteration 2: -44%
+54%

iteration
Total gain: +336% Total gain: -81%
890 executions 1430 executions 180 seconds = 3 minutes 1 minute 44 seconds

➢ Optimizations performed in
From iteration 2:
#12 Sales order (flow A-1) 368 executions From iteration 1: +142% 3 minutes 469 = ~8 minutes From iteration 1: -62% From iteration 2: -42%
+60%

each iteration
Total gain: +289% Total gain: -78%
Submit: 2.0
935 executions 1497 executions Submit: 2.0 seconds End-to-end flow: 9 seconds
seconds
#10 Stock verification 358 executions From iteration 2: 5 seconds End-to-end flow:
From iteration 1: +161% End-to-end flow: 12.0 seconds From iteration 2: -25%
+60% 21.0 seconds
Total gain: +318% From iteration 1: -42% Total gain: -57%
N/A N/A N/A N/A
#9 Price verification 1461 5 seconds End-to-end flow: 45,7 seconds
(run in isolation) (run in isolation) (run in isolation) (run in isolation)

Figure: sample performance benchmark report

Tech Talk: Performance benchmark
- Focus on the performance 1. Define 6. Configure
requirements 7. Execute
performance test
- Keep the objectives reasonable tests
- Document in BRD/FRD objectives environment

- Focus on all critical components 5. Build test 8. Tune and

2. Define
- Keep them close to reality simulations Retest
- Document in BRD/FRD scenarios
and data iteratively

- Keep goals and objectives in

mind 3. Design and
4. Design
- Analyze traces and fix issues develop 9. Report
- Maintain constant checks on tests
system
progress
 Transaction Mix

SO
User1 login
Header
wait Line wait Post

PO
User2 login
Header
wait Line wait Post

GL
User3 login Wait
Header
wait Line wait Post

PO
User4 login wait
Header
wait Line wait Post

SO
User5 login wait
Header
Line Line Line Line

GL
User6 login wait
Header
wait Line wait Post

SO SO
User7 login
Service
wait
Service
wait SO Service
Network/Client

Load pattern Concurrency Pacing

 Narrow it
down

Troubleshoot

Develop the
solution

Test the
solution

Success or
next iteration
 ACTIVITY PARTNER (sample) CUSTOMER (sample)

Define the target/projected business goals I AR

Define the detailed benchmark scenarios RIC A

Take task recordings and document the reproduction steps I AR

Provide the environment artefacts (code build and database to use) I AR

Build the benchmark environment R A

Create test scripts and data scripts R A

Execute the performance benchmark R A

Deliver the performance benchmark report R A

In case of bugs in the standard solution, open a support request to Microsoft C AR

Tune performance (outside Microsoft's scope, e.g. configuration and
R A
customizations)

CAPTION R: Responsible A: Accountable C: Consulted I: Informed

Tech Talk: Performance benchmark
 the environments
planning Tech Talk
Tech Talk: Performance benchmark
1. Define and 6. Conclude and
capture scenarios report

2. Environment 5. Performance test

preparation with incremental
workload

3. Create and
4. Convert to multi-
validate scripts with
user tests
single user
Tech Talk: Performance benchmark
Main article on Docs for PerfSDK
Tech Talk on the performance testing approach
Tech Talk on the troubleshooting tools
Tech Talk on the patterns / anti-patterns
Tech Talk on the RSAT
Considerations for retail performance testing