AWR Data mining

Yury Velikanov Senior Oracle DBA

 Why Companies Trust Pythian
• Recognized Leader:
• Global industry-leader in remote database administration services and consulting for Oracle,
Oracle Applications, MySQL and SQL Server

• Work with over 150 multinational companies such as Forbes.com, Fox Sports, Nordion and Western
Union to help manage their complex IT deployments

• Expertise:
• One of the world’s largest concentrations of dedicated, full-time DBA expertise. Employ 6 Oracle
ACEs/ACE Directors.
• Hold 7 Specializations under Oracle Platinum Partner program, including Oracle Exadata, Oracle
GoldenGate & Oracle RAC.

• Global Reach & Scalability:

• 24/7/365 global remote support for DBA and consulting, systems administration, special projects
or emergency response

2 © 2011 Pythian
Mission

Let you remember/consider AWR

next time you troubleshoot
Performance issue!

3 © 2009/2010 Pythian
NOTE

AWR = STATSPACK = Performance Repository

Excerpt from
11GR2:$OH/rdbms/admin/awrrpt.sql
“Rem This report is based on the Statspack report.”
4 © 2009/2010 Pythian
AWR Agenda

• Introduction & Background

• Examples, Examples, Examples

• Concept & Approach

• More examples

• Q&A
Google: Oracle Yury
Blog, Twitter, Linkedin, ACE … email, phone number

5 © 2009/2010 Pythian
 Few words about Yury
• Oracle ACE and Oracle OCM
- @yvelikanov
- http://www.pythian.com/news/author/velikanov/
• Started as Oracle DBA
- with 7.2 (in 1997, 14+)
• First international appearance
- 2005 - Hotsos Symposium 2005
• Professional Education
- Jonathan Lewis (2003 – 3 days), Tom Kyte (2004 – 3 days), Tanel
Põder (2008 – 2 days ), Cary Millsap (2011 – 3 days) …
• Education (Master Degree in Computer science)
- OCP 7/8/8i/9/10 + OCM 9i/10g/11g

• Oracle DBA consultant experience (14+ years)

• Pythian Oracle Clients support (2+ years)
- 140+ Clients around the world
- Different products, different load, different problems
6 © 2009/2010 Pythian
Background
• AWR is one of many RDBMS performance data sources
Jonathan Lewis / Tom Kyte / Tanel Põder / Cary Millsap
• Sometimes it isn’t the best source (aggregation)
• SQL Extended trace (event 10046)
• RAW trace
• tkprof
• TRCAnlzr [ID 224270.1]
• Method-R state of art tools
• PL/SQL Profiler
• LTOM (Session Trace Collector)
• others

• Sometimes it is the best source!

• Sometimes it is the only one available!

7 © 2009/2010 Pythian
Background
• Once I was called to troubleshoot high load
• Connected to the database I saw 8 active processes
running for 6 hours in average at the time I connected
• I switched 10046 event for all 8 collected 15 minutes
data and analyzed it one by one
• Found several SQLs returning 1 row million times
• Passed the results to development asking to fix the logic
• Spent ~2 hours to find where the issue was

• Next day a workmate asked me

• Why did you use 10046 and spent 2 hours?
• He used AWR report and came up with the same
answer in less than 5 minutes

• Lesson learned: Right tool for the right (job - no) case !
8 © 2009/2010 Pythian
When should you consider AWR mining?
• General resource tuning (high CPU, IO utilization)
• You are asked to reduce server load X times

• You would like to analyze load patterns/trends

• You need to go back in time and see how things progressed

• At System level
• At Session level (ASH)

• Existing official AWR interface doesn’t provide you

information at the right angle/dimension or are not available
(Grid Control, awrrpt.sql)

• You don’t have any other source of information

9 © 2009/2010 Pythian
TOP CPU/IO Consuming SQLs ?
select
s.SQL_ID,
sum(CPU_TIME_DELTA),
sum(DISK_READS_DELTA),
count(*)
from
DBA_HIST_SQLSTAT
group by
SQL_ID
order by
sum(CPU_TIME_DELTA) desc
/

SQL_ID SUM(CPU_TIME_DELTA) SUM(DISK_READS_DELTA) COUNT(*)

------------- ------------------- --------------------- ----------
05s9358mm6vrr 27687500 2940 1
f6cz4n8y72xdc 7828125 4695 2
5dfmd823r8dsp 6421875 8 15
3h1rjtcff3wy1 5640625 113 1
92mb1kvurwn8h 5296875 0 1
bunssq950snhf 3937500 18 15
7xa8wfych4mad 2859375 0 2
...

10 © 2009/2010 Pythian
TOP CPU Consuming SQLs ?

select
s.SQL_ID,
sum(s.CPU_TIME_DELTA),
sum(s.DISK_READS_DELTA),
count(*)
from
DBA_HIST_SQLSTAT s

group by
s.SQL_ID
order by
sum(s.CPU_TIME_DELTA) desc

11 © 2009/2010 Pythian
TOP CPU Consuming SQLs ?
select * from
(
select
s.SQL_ID,
sum(s.CPU_TIME_DELTA),
sum(s.DISK_READS_DELTA),
count(*)
from
DBA_HIST_SQLSTAT s

group by
s.SQL_ID
order by
sum(s.CPU_TIME_DELTA) desc
)
where rownum < 11
/

12 © 2009/2010 Pythian
TOP CPU Consuming SQLs ?
select * from
(
select
s.SQL_ID,
sum(s.CPU_TIME_DELTA),
sum(s.DISK_READS_DELTA),
count(*)
from
DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p
where 1=1
and s.SNAP_ID = p.SNAP_ID

and EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16

group by
s.SQL_ID
order by
sum(s.CPU_TIME_DELTA) desc
)
where rownum < 11
/

13 © 2009/2010 Pythian
TOP CPU Consuming SQLs ?
select * from
(
select
s.SQL_ID,
sum(s.CPU_TIME_DELTA),
sum(s.DISK_READS_DELTA),
count(*)
from
DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p
where 1=1
and s.SNAP_ID = p.SNAP_ID

and EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16

and p.END_INTERVAL_TIME between SYSDATE-7 and SYSDATE

group by
s.SQL_ID
order by
sum(s.CPU_TIME_DELTA) desc
)
where rownum < 11
/

14 © 2009/2010 Pythian
TOP CPU Consuming SQLs ?
select * from
(
select
s.SQL_ID,
sum(s.CPU_TIME_DELTA),
sum(s.DISK_READS_DELTA),
count(*)
from
DBA_HIST_SQLSTAT s, DBA_HIST_SNAPSHOT p, DBA_HIST_SQLTEXT t
where 1=1
and s.SNAP_ID = p.SNAP_ID
and s.SQL_ID = t.SQL_ID
and EXTRACT(HOUR FROM p.END_INTERVAL_TIME) between 8 and 16
and t.COMMAND_TYPE != 47 –- Exclude PL/SQL blocks from output
and p.END_INTERVAL_TIME between SYSDATE-7 and SYSDATE
group by
s.SQL_ID
order by
sum(s.CPU_TIME_DELTA) desc
)
where rownum < 11
/

15 © 2009/2010 Pythian
TOP CPU Consuming SQLs ?

2. 3.
5. 1.

52.8 %
4.

16 © 2009/2010 Pythian
TOP CPU Consuming SQLs ?
select
SQL_ID,
sum(CPU_TIME_DELTA),
sum(DISK_READS_DELTA),
count(*)
from
DBA_HIST_SQLSTAT
group by
SQL_ID
order by
sum(CPU_TIME_DELTA) desc
/

SQL_ID SUM(CPU_TIME_DELTA) SUM(DISK_READS_DELTA) COUNT(*)

------------- ------------------- --------------------- ----------
05s9358mm6vrr 27687500 2940 1
f6cz4n8y72xdc 7828125 4695 2
5dfmd823r8dsp 6421875 8 15
3h1rjtcff3wy1 5640625 113 1
92mb1kvurwn8h 5296875 0 1
bunssq950snhf 3937500 18 15
7xa8wfych4mad 2859375 0 2
...

17 © 2009/2010 Pythian
 5 Slides of
Concept & Approach

18 © 2009/2010 Pythian
AWR = DBA_HIST_% Views +
• 111 Views in 11.2.0.2.0

• I use just few on a regular basis

• DBA_HIST_ACTIVE_SESS_HISTORY - V$ACTIVE_SESSION_HISTORY
• DBA_HIST_SEG_STAT - V$SEGMENT_STATISTICS
• DBA_HIST_SQLSTAT - V$SQL
• DBA_HIST_SQL_PLAN - V$SQL_PLAN
• DBA_HIST_SYSSTAT - V$SYSSTAT ( ~SES~ )
• DBA_HIST_SYSTEM_EVENT - V$SYSTEM_EVENT ( ~SESSION~ )

• Most of the views contain data snapshots from V$___ views

• DELTA columns (e.g. DISK_READS_DELTA)

• DBA_HIST_SEG_STAT
• DBA_HIST_SQLSTAT

19 © 2009/2010 Pythian
AWR Things to keep in mind …
• The data are just snapshots of V$ views

• Data collected based on thresholds

• Some data is excluded based on thresholds

• Some data may not be in SGA at the time of snapshot

• Longer time difference between snapshots more
data got excluded
• For data mining use ALL snapshots available

Begin

End
t
20 © 2009/2010 Pythian
AWR Things to keep in mind …
• Forget about AWR if there are constants in the code
• Indicator is high parse count (hard) (10-50 per/sec)
• It isn’t just hard parsing! (related bugs)
• cursor_sharing = FORCE

• In RAC configuration do not forget INST_ID column in joins

• Most of the V$ (DBA_HIST) performance views have

incremental counters. END - BEGIN values
• You may get wrong results (sometimes negative)
• Sometimes counters reach max value and get reset
• Counters got reset at instance restart time

• Time between snapshots may be different

• Suggestion (ENDv - BEGINv)/(ENDs - BEGINs)=value/sec
21 © 2009/2010 Pythian
AWR Things to keep in mind …
• Seconds count between 2 timestamps
select
s.BEGIN_INTERVAL_TIME,
s.END_INTERVAL_TIME,
s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME DTIME, -- Returns “Interval”
EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) H,
EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) M,
EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) S,

EXTRACT(HOUR FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60*60+

EXTRACT(MINUTE FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME)*60+
EXTRACT(SECOND FROM s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) SECS,

phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) -– Write you own fun()

(cast(s.END_INTERVAL_TIME as date) - cast(s.BEGIN_INTERVAL_TIME as date))
*24*60*60

from
DBA_HIST_SNAPSHOT s
where 1=1
and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE)
and s.DBID = (select DBID from V$DATABASE)
order by
s.BEGIN_INTERVAL_TIME;

22 © 2009/2010 Pythian
AWR Things to keep in mind …
select SNAP_INTERVAL, RETENTION
from
DBA_HIST_WR_CONTROL c, V$DATABASE d
where
c.DBID = d.DBID;

SNAP_INTERVAL RETENTION
------------------------------ ------------------------------
+00000 01:00:00.0 +00007 00:00:00.0

select DBID, INSTANCE_NUMBER, count(*) C,

min(BEGIN_INTERVAL_TIME) OLDEST, max(BEGIN_INTERVAL_TIME) YUNGEST
from
DBA_HIST_SNAPSHOT
group by
DBID,
INSTANCE_NUMBER;

DBID INSTANCE_NUMBER C OLDEST YUNGEST

---------- --------------- ---------- ------------------------- -------------------------
3244685755 1 179 13-AUG-11 07.00.30.233 PM 21-AUG-11 05.00.01.855 AM
3244685755 2 179 13-AUG-11 07.00.30.309 PM 21-AUG-11 05.00.01.761 AM

23 © 2009/2010 Pythian
Trends Analysis Example (1) …
DBA_HIST_SYSSTAT & DBA_HIST_SYSTEM_EVENT

select
s.BEGIN_INTERVAL_TIME, s.END_INTERVAL_TIME,

(
t.VALUE-
LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME)
) DVALUE,

(t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/

phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) VAL_SEC
from
DBA_HIST_SNAPSHOT s,
DBA_HIST_SYSSTAT t
where 1=1
and s.SNAP_ID = t.SNAP_ID
and s.DBID = t.DBID
and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER
and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE)
and s.DBID = (select DBID from V$DATABASE)
and t.STAT_NAME = 'parse count (hard)'
order by
s.BEGIN_INTERVAL_TIME;

24 © 2009/2010 Pythian
Trends Analysis Example (1) …

25 © 2009/2010 Pythian
Trends Analysis Example (1) …
DBA_HIST_SYSSTAT & DBA_HIST_SYSTEM_EVENT

select
s.BEGIN_INTERVAL_TIME, s.END_INTERVAL_TIME,

(
t.VALUE-
LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME)
) DVALUE,

(t.VALUE-LAG (t.VALUE) OVER (ORDER BY s.BEGIN_INTERVAL_TIME))/

phy_get_secs(s.END_INTERVAL_TIME-s.BEGIN_INTERVAL_TIME) VAL_SEC
from
DBA_HIST_SNAPSHOT s,
DBA_HIST_SYSSTAT t
where 1=1
and s.SNAP_ID = t.SNAP_ID
and s.DBID = t.DBID
and s.INSTANCE_NUMBER = t.INSTANCE_NUMBER
and s.INSTANCE_NUMBER = (select INSTANCE_NUMBER from V$INSTANCE)
and s.DBID = (select DBID from V$DATABASE)
and t.STAT_NAME = 'parse count (hard)'
order by
s.BEGIN_INTERVAL_TIME;

26 © 2009/2010 Pythian
SQL Bad performance Example (2) …
• Called by a client to troubleshoot a SQL with bad performance

• Sometimes the SQL hangs (never finishes) and needs to be killed

and re-executed

• Upon re-execution, it always finishes successfully in a few

minutes

• The client demanded a resolution ASAP …

27 © 2009/2010 Pythian
SQL Bad performance Example (2) …
DBA_HIST_SQLSTAT

select
st.SQL_ID
, st.PLAN_HASH_VALUE
, sum(st.EXECUTIONS_DELTA) EXECUTIONS
, sum(st.ROWS_PROCESSED_DELTA) CROWS
, trunc(sum(st.CPU_TIME_DELTA)/1000000/60) CPU_MINS
, trunc(sum(st.ELAPSED_TIME_DELTA)/1000000/60) ELA_MINS
from DBA_HIST_SQLSTAT st
where st.SQL_ID in (
'5ppdcygtcw7p6'
,'gpj32cqd0qy9a'
)
group by st.SQL_ID , st.PLAN_HASH_VALUE
order by st.SQL_ID, CPU_MINS;

28 © 2009/2010 Pythian
SQL Bad performance Example (2) …
DBA_HIST_SQLSTAT

SQL_ID PLAN_HASH_VALUE EXECUTIONS CROWS CPU_MINS ELA_MINS

------------- --------------- ---------- ---------- ---------------- ----------------
5ppdcygtcw7p6 436796090 20 82733 1 3
5ppdcygtcw7p6 863350916 71 478268 5 11
5ppdcygtcw7p6 2817686509 9 32278 2,557 2,765

gpj32cqd0qy9a 3094138997 30 58400 1 3

gpj32cqd0qy9a 1700210966 36 69973 1 7
gpj32cqd0qy9a 1168845432 2 441 482 554
gpj32cqd0qy9a 2667660534 4 1489 1,501 1,642

29 © 2009/2010 Pythian
SQL Bad performance Example (2) …
• In the result …

• Two different jobs were gathering statistics on a daily basis

1. “ANALYZE …” part of other batch job (developer)
2. “DBMS_STATS…” traditional (DBA)

• Sometimes “DBMS_STATS…“ is not completed before the batch

job starts (+/- 10 minutes).

• After the job got killed (typically well after 10 mins since start)
the new “correct” statistics were in place.

• Takeaways …
A. Don’t change your statistics that frequently
B. AWR data helps to spot such issues easily
30 © 2009/2010 Pythian
SQL Bad performance Example (2) …
DBA_HIST_SQLSTAT & DBA_HIST_SEG_STAT

select
st.SQL_ID
, st.PLAN_HASH_VALUE
, sum(st.EXECUTIONS_DELTA) EXECUTIONS
, sum(st.ROWS_PROCESSED_DELTA) CROWS
, trunc(sum(st.CPU_TIME_DELTA)/1000000/60) CPU_MINS
, trunc(sum(st.ELAPSED_TIME_DELTA)/1000000/60) ELA_MINS
from DBA_HIST_SQLSTAT st
where st.SQL_ID in (
'5ppdcygtcw7p6'
,'gpj32cqd0qy9a'
)
group by st.SQL_ID , st.PLAN_HASH_VALUE
order by st.SQL_ID, CPU_MINS;

31 © 2009/2010 Pythian
SQL Plan flipping Example (3) …
• I asked myself: Well !

• If you find that the execution plan for a SQL has changed
from a good (quick) to a bad one (slow), how do you know
if there are other affected SQLs?

• And if there are, how many and which ones?

• Would SQL Profiles (Outlines) help address those?

32 © 2009/2010 Pythian
SQL Plan flipping Example (3) …
SELECT st2.SQL_ID ,
st2.PLAN_HASH_VALUE ,
st_long.PLAN_HASH_VALUE l_PLAN_HASH_VALUE ,
st2.CPU_MINS ,
st_long.CPU_MINS l_CPU_MINS ,
st2.ELA_MINS ,
st_long.ELA_MINS l_ELA_MINS ,
st2.EXECUTIONS ,
st_long.EXECUTIONS l_EXECUTIONS ,
st2.CROWS ,
st_long.CROWS l_CROWS ,
st2.CPU_MINS_PER_ROW ,
st_long.CPU_MINS_PER_ROW l_CPU_MINS_PER_ROW
FROM
(SELECT st.SQL_ID ,
st.PLAN_HASH_VALUE ,
SUM(st.EXECUTIONS_DELTA) EXECUTIONS ,
SUM(st.ROWS_PROCESSED_DELTA) CROWS ,
TRUNC(SUM(st.CPU_TIME_DELTA) /1000000/60) CPU_MINS ,
DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0 , (SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW ,
TRUNC(SUM(st.ELAPSED_TIME_DELTA) /1000000/60) ELA_MINS
FROM DBA_HIST_SQLSTAT st
WHERE 1 =1
AND ( st.CPU_TIME_DELTA !=0
OR st.ROWS_PROCESSED_DELTA !=0)
GROUP BY st.SQL_ID,
st.PLAN_HASH_VALUE
) st2,
(SELECT st.SQL_ID ,
st.PLAN_HASH_VALUE ,
SUM(st.EXECUTIONS_DELTA) EXECUTIONS ,
SUM(st.ROWS_PROCESSED_DELTA) CROWS ,
TRUNC(SUM(st.CPU_TIME_DELTA) /1000000/60) CPU_MINS ,
DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0 , (SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW ,
TRUNC(SUM(st.ELAPSED_TIME_DELTA) /1000000/60) ELA_MINS
FROM DBA_HIST_SQLSTAT st
WHERE 1 =1
AND ( st.CPU_TIME_DELTA !=0
OR st.ROWS_PROCESSED_DELTA !=0)
HAVING TRUNC(SUM(st.CPU_TIME_DELTA)/1000000/60) > 10
GROUP BY st.SQL_ID,
st.PLAN_HASH_VALUE
) st_long
WHERE 1 =1
AND st2.SQL_ID = st_long.SQL_ID
AND st_long.CPU_MINS_PER_ROW/DECODE(st2.CPU_MINS_PER_ROW,0,1,st2.CPU_MINS_PER_ROW) > 2
ORDER BY l_CPU_MINS DESC,
st2.SQL_ID,
st_long.CPU_MINS DESC,
st2.PLAN_HASH_VALUE;

33 © 2009/2010 Pythian
SQL Plan flipping Example (3) …
SELECT
...
FROM
(SELECT st.SQL_ID ,
st.PLAN_HASH_VALUE ,
...
DECODE( SUM(st.ROWS_PROCESSED_DELTA), 0 , 0 ,
(SUM(st.CPU_TIME_DELTA)/1000000/60)/SUM(st.ROWS_PROCESSED_DELTA) ) CPU_MINS_PER_ROW ,
...
FROM DBA_HIST_SQLSTAT st
WHERE 1 =1
...
GROUP BY st.SQL_ID,
st.PLAN_HASH_VALUE
) st2,
(SELECT st.SQL_ID ,
st.PLAN_HASH_VALUE ,
...
HAVING trunc(sum(st.CPU_TIME_DELTA)/1000000/60) > 10
GROUP BY st.SQL_ID,
st.PLAN_HASH_VALUE
) st_long
WHERE 1 =1
AND st2.SQL_ID =
st_long.SQL_ID
AND st_long.CPU_MINS_PER_ROW/DECODE(st2.CPU_MINS_PER_ROW,0,1,st2.CPU_MINS_PER_ROW) > 2
ORDER BY l_CPU_MINS DESC,
st2.SQL_ID,
st_long.CPU_MINS DESC,
st2.PLAN_HASH_VALUE;

34 © 2009/2010 Pythian
 SQL Plan flipping Example (3) …
SQL_ID PLAN_HASH_VALUE L_PLAN_HASH_VALUE CPU_MINS L_CPU_MINS ELA_MINS L_ELA_MINS EXECUTIONS L_EXECUTIONS
------------- --------------- ----------------- ---------- ---------- ---------- ---------- ---------- ------------
db8yz0rfhvufm 3387634876 619162475 17 2673 21 4074 3106638 193
5ppdcygtcw7p6 436796090 2817686509 1 2557 3 2765 20 9
5ppdcygtcw7p6 863350916 2817686509 5 2557 11 2765 71 9
1tab7mjut8j9h 875484785 911605088 9 2112 23 2284 980 1436
1tab7mjut8j9h 2484900321 911605088 6 2112 6 2284 1912 1436
1tab7mjut8j9h 3141038411 911605088 50 2112 57 2284 32117 1436
gpj32cqd0qy9a 1700210966 2667660534 1 1501 7 1642 36 4
gpj32cqd0qy9a 3094138997 2667660534 1 1501 3 1642 30 4
2tf4p2anpwpk2 825403357 1679851684 6 824 71 913 17 13
csvwu3kqu43j4 3860135778 2851322291 0 784 0 874 1 2
0q9hpmtk8c1hf 3860135778 2851322291 0 779 0 867 1 2
2frwhbxvg1j69 3860135778 2851322291 0 776 0 865 1 2
4nzsxm3d9rspt 3860135778 2851322291 0 754 0 846 1 2
1pc2npdb1kbp6 9772089 2800812079 0 511 0 3000 7 695
gpj32cqd0qy9a 1700210966 1168845432 1 482 7 554 36 2
gpj32cqd0qy9a 3094138997 1168845432 1 482 3 554 30 2
* * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * *
4bcx6kbbrg6bv 3781789023 2248191382 0 11 0 41 2 2
6wh3untj05apd 3457450300 3233890669 0 11 0 131 1 20
6wh3untj05apd 3477405755 3233890669 0 11 1 131 2 20
8pzsjt5p64xfu 3998876049 3667423051 0 11 5 44 3 18
bpfzx2hxf5x7f 1890295626 774548604 0 11 0 26 1 24
g67nkxd2nqqqd 1308088852 4202046543 0 11 1 57 1 49
g67nkxd2nqqqd 1308088852 1991738870 0 11 1 39 1 38
g67nkxd2nqqqd 2154937993 1991738870 1 11 27 39 72 38
g67nkxd2nqqqd 2154937993 4202046543 1 11 27 57 72 49

92 rows selected.

Elapsed: 00:00:02.53
SQL>

35 © 2009/2010 Pythian
SQL Plan flipping Example (3) …
• In the result …

• Load on the system was reduced by 5 times

• Performance of batch processing in average was improved by 8

times

• Takeaways …

A. SQL Plans may flip from good plans to …

B. SQL Outlines may help some times
C. AWR provided good input for such analysis

36 © 2009/2010 Pythian
Conclusions …
• AWR = DBA_HIST% views ( snapshots from V$% views )

• Sometimes it is the only source of information

• AWR contains much more information that default AWR reports

and Grid Control could provide you

• Don’t be afraid to discover/mine the AWR data

I can show you the door …

… but it is you who should walk through it

37 © 2009/2010 Pythian
 Pythian Facts
• Founded in 1997, over 14 years
• 100+ employees
• 5 offices in 5 countries (true follow the sun model)
• Employ
• 6 Oracle ACEs (Including 1 ACE director)
• Several Oracle Masters
• Plenty of technical geeks
• Platinum level partner in the Oracle Partner Network
• Actively supports technical communities via
• Blogging
• Conferences
• SIGs and other events

38 © 2009/2010 Pythian
Additional Resources

• www.oracle.com/scan
• www.pythian.com/exadata
• www.pythian.com/news/tag/exadata
Google: Oracle Yury - Exadata
Blog
• www.pythian.com/news_and_events/in_the_news
Article: “Making the Most of Oracle Exadata”

My Oracle Support notes 888828.1 and 757552.1

Thank you!

Blog, Twitter, Linkedin, ACE … email, phone number

39 © 2009/2010 Pythian