You are on page 1of 84

SkillBuilders Presents Kyle Hailey (www.perfvision.

com) Oracle Performance Tuning


Average Session Load (ASL) The Golden Metric ?
Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

SKILLBUILDERS

Who is Kyle Hailey


1990 Oracle
90 Oracle Unix support 92 Ported v6 onto Dec Ultrix 93 Oracle France 94 Gold Support Specialist 95 Performance Expert at Digital Europe 98 Kernel Development California

2000 Dot.Com 2001 Quest 2002 Oracle OEM 10g 2005 Embarcadero 2006 Independent
When not being being a Geek writing for the Oaktable etc - photographer - Just Married and new home owner in Portland - and wonder how I missed the dot.com millions

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

In this Session
The Holy Grail of Performance : ASL 1. ASL : stethoscope for Database Health
Tap is a metric that taps directly into the heart of your DB
Heart stopped - Hung? Beating very slow - Idle ? Beating fast - Heavily Loaded ?

2. ASL Components
CPU Waits Time series

3. Maximum CPU as Yardstick


To measure ASL against

OEM 10g ties it all together


Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Goal of this Presentation


Simplify the data and empower the DBA
My goal is to cut out a lot of the unnecessary information and extraneous data and hone in on the essential When to act How hard to act

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Whats the Database Doing?


Your manager calls, its 2am, do you know where your database is?
Blocked Running ok Idle

I want a quick and easy way to see what the database is doing

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Checking the Database


How do *you* check the Database Health?
Routine Exams? Statspack?
1300 lines of data Which lines Once and hour

Automated Alerts?
What do you set alerts on? What if no alerts fire ? Are you ok? Do alerts really tell you whats happening?
Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Whats the Database Doing?!


Whip out the stethoscope

ASL

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Welcome to The Cult of ASL


Once youve been initiated

there is no going back


For those of you who are already members, welcome back

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Average Session Load (ASL)


The Golden Metric
Powerful Multidimensional Indispensable

Average load of sessions on the database measured in number of sessions

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

10

Core of 10g Performance Monitoring

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Performance Page

11

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

12

Average Session Load (ASL)


aka Average Active Sessions (AAS?)
OEM 10g Graphs

Session Load
I often refer to it this way

Centi-seconds per second (or secs/sec)


In the dark ages, before OEM 10g, waits were often measured as Centi-secs per sec

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

13

ASL Average Session Load


Average
over 15 secs in OEM 10g

Represents Load Active Sessions Only


Active sessions put load the database Inactive Sessions dont put load
Except for memory usage

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

14

ASL : sources
1. Manually
v$system_event v$sysstat

2. Statspack some calculations 3. OEM 10g 4. ASHRPT

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

15

ASL Calculation
ASL = DB TIME / Elapsed Time

DB Time (DBT) = sum over all sessions of time spent in a call. A call could be executing SQL background work DBWR writing blocks LGWR writing redo
Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

DB Time
DB Time (DBT) = sum active session time

16

select value from v$sysstat select value from v$sysstat DB TIME (10g) = where name = DB time; DB time where name = DB time; Select sum(time_waited) from v$system_event Select sum(time_waited) from v$system_event where event not in ((... idle events ); where event not in ... idle events ); + + Select value from v$sysstat Select value from v$sysstat where name = CPU used by this session; where name = CPU used by this session;
Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

DB TIME =

17

1. ASL Manually
Data
10g, easy DBTime = V$sysstat DB time 9i ( 7 to 11 actually), DBTime = V$system_event Sum wait times Non-idle waits V$sysstat CPU used by this session

ASL= DBTime/elapsed time Produces


Session time / elapsed time Session cent-secs/sec Session secs/ sec Avg Session Load (ASL)
Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

18

2. Statspack: Calculating ASL


Look for
Top 5 Timed Events Elapsed Time

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Statspack: Cheat Sheet


Install Connect as SYSDBA @?/rdbms/admin/spcreate.sql Run Exec statspack.snap; Generate Reports @?/rdbms/admin/spreport.sql

19

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

20

Statspack Output
Trusty statspack report Elapased Time Check Top 5 Timed Events

Start at line 52 of about 1300!


Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Statspack Elapsed Time


STATSPACK report for STATSPACK report for DB Name DB Id Instance Inst Num Release RAC DB Name DB Id Instance Inst Num Release RAC Host Host ------- ----------- -------- -------- ---------- --- ------------- ----------- -------- -------- ---------- --- ------LABSF03 1420044432 1 10.1.0.2.0 NO labsfr LABSF03 1420044432 labsf03 labsf03 1 10.1.0.2.0 NO labsfr Snap Id Snap Time Sessions Curs/Sess Snap Id Snap Time Sessions Curs/Sess --------- ------------------ -------- ----------------- ------------------ -------- --------Begin Snap: 1 03-Apr-06 12:34:06 18 5.6 Begin Snap: 1 03-Apr-06 12:34:06 18 5.6 End Snap: 2 03-Apr-06 12:34:36 18 4.8 End Snap: 2 03-Apr-06 12:34:36 18 4.8 Elapsed: 1.00 (mins) Elapsed: 1.00 (mins)

21

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

22

Statspack: Top 5 Timed Events


Look at Top 5 Timed Events
Top 5 Timed Events Top 5 Timed Events ~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~ Event Waits Event Waits Time (s) Time (s) --------------------- --------- ----------------------------- --------- --------buffer busy waits 2,748 250 buffer busy waits 2,748 250 CPU time 32 CPU time 32 free buffer waits 1,588 15 free buffer waits 1,588 15 write complete waits 10 8 write complete waits 10 8 log buffer space 306 5 log buffer space 306 5 % Total % Total Call Time Call Time ------------78.72 78.72 10.16 10.16 4.63 4.63 2.51 2.51 1.51 1.51

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Example
Top 5 Timed Events Top 5 Timed Events

23

CPU + WAITS
CPU = 32 WAITS = 250 + 15 + 8 + 5 = 278 secs

Elapsed Time
= 60 secs

Event Time (s) Event Time (s) --------------------------------- --------buffer busy waits 250 buffer busy waits 250 CPU time 32 CPU time 32 free buffer waits 15 free buffer waits 15 write complete waits 8 write complete waits 8 log buffer space 5 log buffer space 5

(32 + 278) secs / 60 secs = 5.1 Average Session Load

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

3. ASL in OEM DB Home Page

24

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

4. ASHRPT
?@/rdbms/admin/ashrpt.sql

25

Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Avg Active Avg Active Event Class % Activity Event Class % Activity Sessions Sessions --------------------------- ----------- ---------- ------------------------------------ ----------- ---------- ---------CPU + Wait for CPU CPU 67.98 8.42 CPU + Wait for CPU CPU 67.98 8.42 enq: TX - row lock co Application 23.98 2.97 enq: TX - row lock co Application 23.98 2.97 buffer busy waits Concurrency 4.66 0.58 buffer busy waits Concurrency 4.66 0.58 latch: cache buffers chai 2.26 0.28 latch: cache buffers chai Concurrency Concurrency 2.26 0.28

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

26

Got ASL, Now What ?


Need one more item: CPU Count # of CPUs available on System
Shared with other applications ? difficult On dual & quad cores, lower the CPU count Represents max active sessions doing work
A value above CPU represents some bottleneck

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

CPU count, OEM DB Home Page

27

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

28

CPU Count, OEM Perf Page

CPU Count
Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

29

Available CPU init.ora


Statspack 10g shows # of CPUS Statspack 9i # of CPUs missing:
SQLPLUS> show parameters cpu_count SQLPLUS> show parameters cpu_count NAME VALUE NAME VALUE ------------------ --------------------------- ---------cpu_count 2 cpu_count 2

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

30

4. ASHRPT
ASH Report For TESTDB/testdb ASH Report For TESTDB/testdb TESTDB/testdb DB Name DB Id Instance Inst Num Release RAC Host DB Name DB Id Instance Inst Num Release RAC Host ------------ ----------- ------------ -------- ----------- --- ----------------------- ----------- ------------ -------- ----------- --- -----------TESTDB 2371570538 testdb 11 10.2.0.1.0 NO sdbe604a TESTDB 2371570538 testdb 10.2.0.1.0 NO sdbe604a CPUs SGA Size Buffer Cache Shared Pool ASH Buffer Size CPUs SGA Size Buffer Cache Shared Pool ASH Buffer Size ---- ------------------ ------------------ ------------------ --------------------- ------------------ ------------------ ------------------ -----------------22 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) 1,000M (100%) 468M (46.8%) 112M (11.2%) 4.0M (0.4%) Analysis Begin Time: 21-Apr-06 12:00:01 21- AprAnalysis Begin Time: 21-Apr-06 12:00:01 Analysis End Time: 21-Apr-06 12:05:01 21- AprAnalysis End Time: 21-Apr-06 12:05:01 Elapsed Time: 5.0 (mins) mins) Elapsed Time: 5.0 ((mins) Sample Count: 3,716 Sample Count: 3,716 Average Session Load: 12.39 Average Session Load: 12.39 Avg. Active Session per CPU: 6.19 Avg. Active Session per CPU: 6.19 Report Target: None specified Report Target: None specified Top User Events DB/Inst: TESTDB/testdb (Apr 21 12:00 to 12:05) Top User Events DB/Inst: TESTDB/testdb TESTDB/testdb (Apr 21 12:00 to 12:05) Avg Active Avg Active Event Event Class %% Activity Event Event Class Activity Sessions Sessions ----------------------------------- --------------- ---------- -------------------------------------------- --------------- ---------- ---------CPU ++ Wait for CPU CPU 67.98 8.42 CPU Wait for CPU CPU 67.98 8.42 enq: TX -- row lock contention Application 23.98 2.97 enq: TX enq: row lock contention Application 23.98 2.97 buffer busy waits Concurrency 4.66 0.58 buffer busy waits Concurrency 4.66 0.58 latch: cache buffers chains Concurrency 2.26 0.28 latch: cache buffers chains Concurrency 2.26 0.28 Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

ASL Formulas
Use CPU count as yardstick: <1
Database is not blocked

31

ASL ~= 0
Database basically idle

ASL < # of CPUs


CPU available Database is probably not blocked

ASL > # of CPUs


Could have performance problems

ASL >> # of CPUS


There is a bottleneck

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Available CPU vs ASL


ASL = 5.1 ASL = 5.1 # of CPU = 2 # of CPU = 2
ASL far above available CPU => problem

32

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

33

Going Farther with ASL


ASL can tell you a lot But its components tell you much more To go farther need the components of ASL
CPU
How many CPUs (max CPU available)

Wait
Which waits

Value over time

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

EM DB Home Page

34

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

OEM 10g Perf Pages


DB Home

35

ASL Point in Time

Performance

ASL over Time

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Performance Page

36

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Performance Page

37

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

38

ASL Performance Page

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

39

OEM 10g - ASL


ASL is the top of the curve

1.9

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

40

OEM 10g - CPU

CPU Usage
Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

41

OEM 10g - Waits

Waits

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

42

OEM 10g CPU vs Wait

CPU

Waits

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

43

OEM 10g Max CPU

Maximum Possible CPU


Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

44

OEM 10g Zoom-In

CPU + WAIT Available CPU


Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

45

OEM 10g get to work !

Relax

Get to Work!

Looks OK But

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

46

Alternative ASL Calculations


ASL = DB TIME / Elapsed Time
But there is another way

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

47

ASL based on ASH


ASH - Active Session History
v$active_session_history

Formula
Count active sessions over 15 seconds Divided by # of samples in 15 seconds

= Average Session Load


A statistical approximation, but surprisingly close

ASH data source empowers drilldown into the causes of bottlenecks


Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

48

2. ASL from ASH


2
7 6 5 4 3 2 1 7 6 5 4 3 2 1

ASH

t0-n

t0

tn

ASL =

active sessions(ti) / n
n 0 Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

49

Two Sources comparison


1. v$system_event & v$sysstat
Indirect Accurate Lags (especially CPU) Limits analysis

2. v$active_session_history
Direct Real time Approximation Allows drilldowns
Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

OEM 10g Perf Pages


DB Home

50

Performance

1 Same Chart but


Top Activity

calculated differently

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

DB Home

1. ASL Performance Page

51

Performance

Top Activity

v$sysstat

v$system_event

ASL = DBT / Time

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

52

2. ASL Top Activity Page


DB Home Performance

Top Activity

v$active_session_history ASL =

active sessions(ti)
n 0

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

53

Samples VS Counters
Samples v$active_session_history Slight Lags

v$system_event

Counters

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

The Power ASH gives ASL


DB Home

54

Performance

Top Activity ASH

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

ASH in OEM

55

Top Activity gives more information


Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

56

Top Activity : Based on ASH


missing

Thanks To ASH

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

57

Top Activity : ASH Dimensions

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

ASL %Session Time Issue

58

Shown in % DB Time Missing % Session Time


Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Top Activity: ASH Sessions

59

Many Users Active

On Performance Page, no way to tell how many users But Top Activity Page fixes that
Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Top Activity: ASH Sessions

60

Two Users Active

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

OEM 10g Perf Pages


DB Home

61

Performance

Top Activity Session SQL

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

OEM 10g Perf Pages


Top Activity

62

SQL

Session

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Session : ASH Activity

63

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

SQL : ASH Activity

64

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

65

Getting the Most out of ASL


Need to know the Systems Profile What your application is like?
Data Warehouse OLTP Typical load

Once you get to know it you can see anomalies


Is ASL near 0 when it should be higher Is that Data Warehouse query running normal
Do you know what it looks like?

Is there an unusual bottleneck

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Knowing your DB Profile

66

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

67

When to tune?
General rules of Thumb Waits >> CPU CPU > Max CPU

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

68

Waits > CPU

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

CPU > Max CPU

69

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

70

Idle Database
Value of proving the database is Idle Its the Databases Fault
How many times do you hear that?

Database Idle
No load on database Database performance is fine
Under utilized

Problem lies elsewhere Saved me time and stress many times


Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

71

Idle Database Perf Page

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Idle Database Top Activity

72

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Harnessing ASL
Statspack
lacking

73

OEM 10g
Best tool 10g (or 11) only Need licences

Third Party Tools


Some good products Generally as expensive as OEM

Ashmon OEM-lite
Free Any version with S-ASH Limited analysis
Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

74

Statspack
Free and any version Needs numerical massaging Lacking detailed SQL and Session info

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

OEM 10g
Clearest, easiest, most powerful
Overview
Load chart - ASL

75

Breakdown
Top SQL and Session Or Service, Module, Action, Objects, Files

Drill downs
SQL Session

Plus
ADDM Automatic Database Diagnostics Monitor SQL Advisor SQL tuning advice

DRAWBACK 10g (or 11) and need license

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

OEM 10g Perf Pages


DB Home

76

Performance

Top Activity Session

SQL

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

77

Alternatives
Third Party tools
Quest Performance Analyzer Symantec I3 Confio Ignite

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

78

ASHMON a free OEM-lite


http://perfvision.com/ashmon.php Windows application Monitors Oracle 10g or 11 Show Top Activity page from OEM Note: Can be used on any version of Oracle with performance repository available on http://perfvision.com/sash.php
Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

CPU > Max CPU

79

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

Radical Change in Profile

80

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

81

Going Further Wait Analysis


If ASL identifies Wait Bottlenecks Use OEM 10g
Automatic Diagnostic Database Monitor (ADDM)

Oracle Wait Interface


by R Shee and K Deshpande

Wait Interface Class


Detail explanation, analysis and solutions to the top 30 wait events (98% of all waits encountered)

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

82

Where Can You Learn More


Oracle 10g Advanced Performance Tuning Class
With Kyle Hailey and SkillBuilders

2 Day Class Offsite or onsite Registration


www.skillbuilders.com Gary Belke at 1-888-803-5607

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

83

Oracle 10g Advanced Performance Tuning Class


Learn new 10g performance and diagnostic tools Understand the 10g Enterprise Manager Performance page ASL, AWR, ADDM Advanced level of detail
Collection, Aggregation, Display, Analyze

Top 30 wait events ASH Conquer your database woes!

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.

84

In summary
ASL is simple and Powerful ASLs components are even more powerful
CPU WAIT Value over Time

Use # of CPUs as a yardstick Know your application load profile to see anomalies Monitor ASL with OEM 10g, Ashmon or third party tools

Copyright 2007-2007 Kyle Hailey. Distributed by www.skillbuilders.com.