Performance Testing

and OBIEE

Robin Moffatt, WM Morrisons plc

http://rnm1978.wordpress.com

Introduction
Oracle
Big

BI specialist at Morrisons plc

IT development programme at its

early stages implementing OBIEE,
OBIA, ORDM, all on Oracle 11g & HPUX

The aim of this

presentation
A

Performance Tuning Methodology

OBIEE

techie stuff

Learn

from my mistakes!

What is performance testing all

about?
Response

times

Report
ETL batch
OLTP transaction

System

impact

Resource usage
Scalability

Why performance test?

(Isnt testing just for wimps?)

Check

that your system performs

Are the users going to be happy?

Baseline
How fast is fast?
How slow is slow?

Validate

system design

Do it right, first time

Capacity

planning

Why performance test?

Its

never too late

Youll never catch all your

problems in pre-production testing.

Thats why you need a reliable and
efficient method for solving the
problems that leak through your preproduction testing processes.
Cary Millsap - Thinking Clearly About Performance

Why performance test?

Because

it makes you better at your job

At the very least, your performance test

plan will make you a more competent

diagnostician (and clearer thinker)
when it comes time to fix the performance
problems that will inevitably occur
during production operation.
Cary Millsap - Thinking Clearly About Performance

Performance Testing What

& Why
Quantifying

response times

System impact

User expectations
Problem diagnosis
Design validation

Performance Testing How?

Define
Do it right
Timebox!
Evaluate
design /
config options
Redefin
e test

Be
Methodical

Measure

Dont fudge it

Analyse

Do more
testing

Review

Do more
testing

Impleme
nt

Iterati
ve
approa
ch

Define
Define
Measure
Measure

Define & build your test

Analyse
Analyse
Review
Review
Impleme
Impleme
nt
nt

Define

what are you going to test

Aim of the test

Scope
Assumptions
Specifics

E.g. :
Check that the system
performs
Baseline performance
Prove system capacity
Validate system design

Data, environment, etc

Build

how are you going to test it

OBIEE specific

Define
Define
Measure
Measure

Consider your test scope

Analyse
Analyse
Review
Review

Impleme
Impleme
nt
nt

Fewer
More components
components =
= easier
more complex
to manage
= more
= more
variables
precise==
larger
more
margin
efficient
of error

Define
Define
Measure
Measure

OBIEE stack

Analyse
Analyse
Review
Review
Impleme
Impleme
nt
nt

Report /
Dashboard
Rendere
d report

Presentation
Services

Data
set

Logical SQL

BI Server

Physical
SQL
statement(s
)
Data
set(s)

Excludes App/Web server &

presentation services plug-in

Database

Define
Define
Measure
Measure

OBIEE testing options

Analyse
Analyse
Review
Review
Impleme
Impleme
nt
nt

User &
Stopwatc
h

Load Testing
tool
(eg.
LoadRunner,
OATS)

ed
er t
nd or
Re rep

Presentation
Services

LS
Q
L
Da
t

se

nqcmd

LSQL
Physic
al
SQL

BI Server
Da
t

se
t

(s
)

SQL
Client

Physical
SQL

Database

Define
Define
Measure
Measure

OBIEE testing options

Analyse
Analyse
Review
Review
Impleme
Impleme
nt
nt

nqcmd

LSQL
Physic
al
SQL

BI Server
Da
t

se
t

(s
)

Database

Define
Define
Measure
Measure

nqcmd

Analyse
Analyse
Review
Review
Impleme
Impleme
nt
nt

Usage
Tracking or
NQQuery.log

Test
Test
script
script

Logical
SQL

Logical
SQL Logical
SQL
nqcmd

BI
Serve
r

Data

Define
Define
Measure
Measure

nqcmd

Analyse
Analyse
Review
Review

Test
Test
scri
scri
pt
pt
Test
Test
scri
scri
pt
pt

nqcmd

Test
Test
scri
scri
pt
pt

nqcmd

nqcmd

Test
Test
scri
scri
pt
pt

Logical
SQL

Master
Master test
test script
script

Impleme
Impleme
nt
nt

nqcmd

BI
Serve
r

Data

Define
Define

LoadRunner
a.k.a. HP Performance Centre
Simulates
Powerful,

user interaction HTTP traffic

but can be difficult to set up

Ajax complicates things

Do

you really need to use it?

Tools
Fiddler2
FireBug

Reference:
My Oracle Support Doc ID 496417.1
http://rnm1978.wordpress.com/category/loadrunner

Measure
Measure
Analyse
Analyse
Review
Review
Impleme
Impleme
nt
nt

Defining your test summary

Be

very clear what the aim of your

test is
You probably need to define multiple
tests
Different points on the OBIEE stack
to interface
Pick the most appropriate one

Write

everything down!

Define
Define
Measure
Measure
Analyse
Analyse
Review
Review
Impleme
Impleme
nt
nt

Define

Measure

Analyse

Review

Impleme
nt

Define
Define

OBIEE measuring &

monitoring
Enterprise
Manager
(Oracle)

Apache
log
OAS log
Analytics
log

Web
Server
App
Server

Presentation
Services plugin

sawserver.lo
g

Presentation
Services
PerfMon
(windows
only)
jConsole
etc

Server metrics
e.g. : IO, CPU,
Memory

NQServer.log

Usage
Trackin
g

NQQuery.log

BI Server

Measure
Measure
Analyse
Analyse
Review
Review
Impleme
Impleme
nt
nt

PerfMon
(Windows)
Oracle OS
Watcher
(unix)

Enterprise
Manager
ASH, AWR,
SQL
Monitor

systems
management

Database
Enterprise
Manager

BI
Management
BI Management
Pack
Pack

Presentati
on
services

Define
Define
Measure
Measure

Oracle SQL Monitor

Analyse
Analyse
Review
Review
Impleme
Impleme
nt
nt

Define
Define
Measure
Measure

Measure - summary

Analyse
Analyse
Review
Review
Impleme
Impleme
nt
nt

Lots

of different ways to measure

Build

measurement into your test

plan
Automate where possible
Easier
Less error

Define

Measure

Analyse

Review

Impleme
nt

Define
Define
Measure
Measure

Analysing the data

Analyse
Analyse
Review
Review
Impleme
Impleme
nt
nt

Define
Define
Measure
Measure

Analysing the data

Analyse
Analyse
Review
Review
Impleme
Impleme
nt
nt

Define
Define
Measure
Measure

Analysing the data

Analyse
Analyse
Review
Review
Impleme
Impleme
nt
nt

Response
time
1
1
9
1
3
2
2
10
2
2
3
1
3
2
9
3
10

Average
(mean)
3.4

50th
percentile
(Median)
2
90th
percentile
9.1

Recording data about

the test

Define
Define
Measure
Measure
Analyse
Analyse
Review
Review
Impleme
Impleme
nt
nt

Dashboard
Requests

SQL IDs
Execution plan hash id

Physical
SQL
Execution
plan

ORA_HASH(QUERY_TEXT
)

Logical SQL

Extending Usage
Tracking
S_NQ_ACCT
START_TS

OBIEE_REPLAY_STATEMENTS
qt_ora_hash
query_text
saw_path
dashboard

ROW_COUNT
TOTAL_TIME_SEC
NUM_DB_QUERY
QUERY_TEXT
QUERY_SRC_CD
SAW_SRC_PATH
SAW_DASHBOARD

OBIEE_REPLAY_STATS
testid
testenv
qt_ora_hash
start_ts
response_time
row_count
db_query_cnt

Define
Define
Measure
Measure
Analyse
Analyse
Review
Review
Impleme
Impleme
nt
nt

Analyse
Define
Do it right
Measure
Evaluate
design /
config options

Timebo
x!

Analyse

Review

Impleme
nt

Dont fudge it

Iterati
ve
approa
ch

Review
Define

Measure
Continu
e
testing

Redefin
e test

Analyse

Review
Implement
Impleme
nt

Iterati
ve
approa
ch

Define
Define
Measure
Measure

Review

Analyse
Analyse
Review
Review
Impleme
Impleme
nt
nt

Implement
Define

Measure

Analyse

Review

Impleme
nt

Iterati
ve
approa
ch

Lessons Learnt
You

wont get your testing right first time

Theres no shame in that

Dont cook the books
Better to redefine your test than invalidate its results

Stick

to the methodology

Dont move the goalposts

Very tempting to pick off the low-hanging fruit
If you do, make sure you dont get indigestion

Timebox
Test

your implementation!

Performance Testing
OBIEE
Define
Do it right
Dont fudge it

Measure
Evaluate
design /
config options
Redefin
e test

Be
Methodical

robin@moffatt.me

Analyse

Do more
testing

Review

Do more
testing

Iterati
ve
approa
ch

Impleme
nt
http://rnm1978.wordpress.com

http://twitter.com/rnm1978

#EOF