<Insert Picture Here>

Analyzing SQL Traces generated by EVENT 10046

Carlos Sierra
Consulting Technical Advisor
 Presentation Agenda
• Event 10046 SQL Trace components
• Recursive and Non-recursive SQL <Insert Picture Here>

• Database Calls
• Idle and Non-Idle Waits
• Binds
• Analyzing the 10046 SQL Trace with TKPROF
• Making sense of User Response Time

2
 Event 10046 SQL Trace

• Generates a SQL Trace for process performance

analysis
• Levels
• 1: Standard SQL Trace
• 4: Includes Bind Variables
• 8: Includes Waits
• 12: Includes Bind Variables and Waits

3
 Tracing with Event 10046

ALTER SESSION SET EVENTS '10046 TRACE NAME CONTEXT FOREVER, LEVEL 12';

DECLARE
cust_type NUMBER;
over_limit NUMBER;
BEGIN
FOR i IN 1 .. 5
LOOP
select
max(orders_total - credit_limit)
into over_limit
from customer_v v, part
where orders_total > credit_limit
and customer_type = i;
END LOOP;
END;
/

QUIT;

4
Event 10046 SQL Trace sample
=====================
PARSING IN CURSOR #2 len=122 dep=1 uid=119 oct=3 lid=119 tim=1249721241739419 hv=670147313 ad='9820f690' sqlid='6krw4hcmz38rj'
SELECT MAX(ORDERS_TOTAL - CREDIT_LIMIT) FROM CUSTOMER_V V, PART WHERE ORDERS_TOTAL > CREDIT_LIMIT AND CUSTOMER_TYPE = :B1
END OF STMT
BINDS #2:
Bind#0
oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
oacflg=03 fl2=206001 frm=00 csi=00 siz=24 off=0
kxsbbbfp=b7d92144 bln=22 avl=02 flg=05
value=5
EXEC #2:c=0,e=0,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=1,plh=1302937547,tim=1249721241739419
FETCH #2:c=189971,e=197994,p=0,cr=3065,cu=0,mis=0,r=1,dep=1,og=1,plh=1302937547,tim=1249721241937413
CLOSE #2:c=0,e=0,dep=1,type=3,tim=1249721241937413
EXEC #5:c=1834721,e=1847934,p=1057,cr=17116,cu=0,mis=0,r=1,dep=0,og=1,plh=0,tim=1249721241937413
WAIT #5: nam='SQL*Net message to client' ela= 3 driver id=1413697536 #bytes=1 p3=0 obj#=92439 tim=1249721241942899

*** 2009-08-08 04:47:22.066

WAIT #5: nam='SQL*Net message from client' ela= 123764 driver id=1413697536 #bytes=1 p3=0 obj#=92439 tim=1249721242066733
CLOSE #5:c=0,e=0,dep=0,type=0,tim=1249721242047436
WAIT #0: nam='SQL*Net message to client' ela= 3 driver id=1413697536 #bytes=1 p3=0 obj#=92439 tim=1249721242066918
WAIT #0: nam='SQL*Net message from client' ela= 123471 driver id=1413697536 #bytes=1 p3=0 obj#=92439 tim=1249721242190411
XCTEND rlbk=0, rd_only=1, tim=1249721242179454
=====================

5
 Recursive and Non-Recursive SQL

• User SQL • User SQL

• User SQL • Recursive SQL
• User SQL • Recursive SQL
• Commit • User SQL
• User SQL
• Recursive SQL (dep 1)
• Recursive SQL (dep 2)
Non-Recursive = (depth = 0)
• Recursive SQL
Recursive (depth > 0)
• Recursive SQL
• Recursive SQL
• Commit

6
 Database Calls

• User SQL • User SQL

• Recursive SQL • Parse
• Recursive SQL • Exec
• Recursive SQL
• Parse
• Exec
• Fetch
Common DB Calls: • Recursive SQL
Parse, Exec and Fetch • Exec
• Fetch
• Exec
• Fetch
• Fetch

7
 Recursive SQL in action

=====================
PARSING IN CURSOR #3 len=210 dep=2 uid=0 oct=3 lid=0 tim=122026736875151
select /*+ rule */ bucket_cnt, row_cnt, cache_cnt, null_cnt, timestamp#,
END OF STMT
EXEC #3:c=0,e=18,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=3,tim=1220267368751515
FETCH #3:c=0,e=23,p=0,cr=4,cu=0,mis=0,r=1,dep=2,og=3,tim=122026736875159
STAT #3 id=1 cnt=7 pid=0 pos=1 obj=255 op='TABLE ACCESS BY INDEX ROWID H
STAT #3 id=2 cnt=7 pid=1 pos=1 obj=257 op='INDEX RANGE SCAN I_HH_OBJ#_IN
EXEC #1:c=4000,e=3607,p=1,cr=29,cu=0,mis=1,r=0,dep=1,og=4,tim=1220267368
FETCH #1:c=1000,e=5188,p=1,cr=2,cu=0,mis=0,r=1,dep=1,og=4,tim=1220267368
=====================

8
 Idle and Non-Idle Waits

• SQL • SQL
• Parse • Parse
• Exec • Exec
• Fetch • Fetch
• Fetch • db file sequential read
• SQL • db file sequential read
• SQL*Net message to client
Non-Idle = within a db call • Fetch
Idle = between db calls • SQL*Net message from client
• SQL

9
 Recursive and Non-Recursive SQL

SQL depth 0
SQL SQL

depth 1

SQL SQL SQL SQL

depth 2
SQL Genealogy represents the
relationship between SQL
SQL
statements within a process

Time

10
Database Calls

SQL depth 0
SQL SQL

FETCH PARSE EXEC FETCH FETCH PARSE

depth 1
SQL

EXEC FETCH

Time

11
Database Calls and Waits
SQL depth 0 SQL SQL

FETCH Idle Idle

PARSE EXEC FETCH FETCH PARSE
Wait Wait

Non-Idle Non-Idle Non-Idle

Non-Idle Non-Idle
Wait Wait Wait
Wait Wait

SQL depth 1

EXEC FETCH

Time

12
 Concepts Summary
• SQL can be classified as:
• Non-Recursive SQL (depth = 0)
• Recursive SQL (depth > 0)
• User SQL (depth  0)
• Internal SQL (depth > 0)
• Most common Database Calls:
• Parse - CBO Produces an Execution Plan
• Execute - DML operations
• Fetch - Queries
• Waits can be classified as:
• Non-Idle Waits (within DB Calls)
• Idle Waits (between DB Calls)

13
 <Insert Picture Here>

Analyzing the 10046

SQL Trace with TKPROF

14
 Motivation

• End users care about response time (clock time)

• 10046 analysis driver must be user response time
• Analyzing an Event 10046 Trace manually is hard
• Several tools can help to digest Event 10046 Trace
• TKPROF is the most commonly used

15
TKPROF time summary
OVERALL TOTALS FOR ALL NON-RECURSIVE STATEMENTS

call count cpu elapsed disk query current rows

------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.01 0.02 15 116 0 0
Execute 1 1.83 1.84 1057 17116 0 1
Fetch 0 0.00 0.00 0 0 0 0
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 2 1.84 1.86 1072 17232 0 1

Elapsed times include waiting on following events:

Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
SQL*Net message to client 4 0.00 0.00
SQL*Net message from client 4 0.12 0.49
db file sequential read 2 0.00 0.00
log file sync 1 0.00 0.00

OVERALL TOTALS FOR ALL RECURSIVE STATEMENTS

call count cpu elapsed disk query current rows

------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 68 0.00 0.00 0 0 0 0
Execute 208 0.12 0.13 6 41 2 1
Fetch 320 1.71 1.73 1064 17190 0 1065
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 596 1.84 1.86 1070 17231 2 1066

Elapsed times include waiting on following events:

Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file sequential read 58 0.00 0.00
db file scattered read 83 0.00 0.00

16
 TKPROF sample for one SQL
SQL ID: 6krw4hcmz38rj
Plan Hash: 1302937547
SELECT MAX(ORDERS_TOTAL - CREDIT_LIMIT)
FROM
CUSTOMER_V V, PART WHERE ORDERS_TOTAL > CREDIT_LIMIT AND CUSTOMER_TYPE = :B1

call count cpu elapsed disk query current rows

------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.00 0.00 0 0 0 0
Execute 5 0.06 0.04 6 40 0 0
Fetch 5 1.24 1.23 463 15325 0 5
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 11 1.31 1.27 469 15365 0 5

Rows Row Source Operation

------- ---------------------------------------------------
1 SORT AGGREGATE (cr=3065 pr=463 pw=0 time=0 us)
100000 MERGE JOIN CARTESIAN (cr=3065 pr=463 pw=0 time=43985 us cost=7014 size=421169364 card=11699149)
5 HASH JOIN (cr=3019 pr=420 pw=0 time=0 us cost=293 size=21060 card=585)
1281 JOIN FILTER CREATE :BF0000 (cr=81 pr=78 pw=0 time=0 us cost=25 size=20000 card=2000)
1281 TABLE ACCESS FULL CUSTOMER (cr=81 pr=78 pw=0 time=0 us cost=25 size=20000 card=2000)
267 VIEW (cr=2938 pr=342 pw=0 time=0 us cost=267 size=321516 card=12366)
267 HASH GROUP BY (cr=2938 pr=342 pw=0 time=0 us cost=267 size=370980 card=12366)
298 VIEW VM_NWVW_1 (cr=2938 pr=342 pw=0 time=0 us cost=267 size=370980 card=12366)
298 HASH GROUP BY (cr=2938 pr=342 pw=0 time=0 us cost=267 size=2028024 card=12366)
1662 JOIN FILTER USE :BF0000 (cr=2938 pr=342 pw=0 time=87993 us cost=264 size=2028024 card=12366)
10873 HASH JOIN (cr=2938 pr=342 pw=0 time=87993 us cost=264 size=2028024 card=12366)
20000 TABLE ACCESS FULL PART (cr=145 pr=142 pw=0 time=22004 us cost=44 size=200000 card=20000)
10873 HASH JOIN (cr=2793 pr=200 pw=0 time=65996 us cost=220 size=1904364 card=12366)
1959 NESTED LOOPS (cr=2167 pr=34 pw=0 time=22002 us cost=42 size=232254 card=2277)
2034 TABLE ACCESS FULL SALES_ORDER (cr=131 pr=12 pw=0 time=22002 us cost=41 size=195822 card=2277)
1959 INDEX UNIQUE SCAN CUSTOMER_PK (cr=2036 pr=22 pw=0 time=0 us cost=0 size=16 card=1)(object id 92435)
164802 TABLE ACCESS FULL ORDER_LINE (cr=626 pr=166 pw=0 time=21998 us cost=175 size=7047924 card=135537)

Elapsed times include waiting on following events:

Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
db file sequential read 19 0.00 0.00
db file scattered read 63 0.00 0.00

17
 Beyond TKPROF

• TKPROF is a good starting point

• Identifies expensive SQL
• Provides Row Source Plan and Explain Plan
• TKPROF has some limitations
• Time Summary is sometimes hard to understand
• Limited content for Row Source Plan and Explain Plan
• Silent about Bind Variables
• No relationships between Recursive and Non-Recursive SQL
• May core-dump or die on large traces
• No details about accessed objects (tables and indexes)
• Text mode only (poor readability)

18
 <Insert Picture Here>

Making sense of
User Response Time

19
 User Response Time Analysis
Unaccounted-for Time
Idle Wait Time

DB Call ET DB Call ET
User Response Time

Time

• Database Calls: PARSE, EXEC, FETCH, etc.

• Total User Response Time = DB Calls Elapsed Time
+ Idle Wait Times + Unaccounted-for Response Time
• Accounted-for RT = Elapsed Time + Idle Wait Time

20
 Response Time

Database Call Elapsed Time

Unaccounted-for
Response Time
Idle Wait Time

Accounted-for
Response Time

21
 Database Call Analysis
Unaccounted-for Time
Database Call CPU Time

OS Call ET OS Call ET
Database Call Elapsed Time

Time

• OS Call Elapsed Time = Non-Idle Wait Time

• DB Call Elapsed Time = DB Call CPU Time + Non-
Idle Wait Times + Unaccounted-for Elapsed Time
• Accounted-for ET = CPU + Non-Idle Wait Time

22
 Database Call Elapsed Time

Database Call CPU Time

Unaccounted-for
Elapsed Time
Non-Idle Wait Time

Accounted-for
Elapsed Time

23
 Response Time Accounted-for
Elapsed Time
Database Call
Elapsed Time

Database Call CPU Time

Unaccounted-for
Elapsed Time
Non-Idle Wait Time

Unaccounted-for
Response Time
Idle Wait Time

Accounted-for
Response Time

24
Response Time
Accounted-for Time

Database Call CPU Time

Non-Idle Wait Time

Idle Wait Time Unaccounted-for

Time

25
 Unaccounted-for Time

• Process not executing (waiting in OS ready queue)

• Trace of Waits not requested (10046 level 8 or 12)
• Writing into a SQL Trace file
• Visiting a non-instrumented event
• Others (usually small)
• Measurement instrumentation overhead (time record keeping)
• Quantization error (CPU measured in CS, not in micro-secs)
• Double-counting CPU (negative unaccounted-for time)
• Counted twice: by DB Call and by OS Call from DB Call

26
Response Time Aggregation
* *
Database Calls Non-Idle Unaccounted-for
Elapsed Time
CPU Time Wait Time

* *
Database Calls Idle Unaccounted-for
Response Time
Elapsed Time Wait Time

*
Response Time

(*) from trace

27
 Response Time Summary
• Response Time Components
• Elapsed Time (from DB Calls)
• CPU Time (from DB Calls)
• Non-Idle Wait Time (within DB Calls)
• ET Unaccounted-for
• Idle Wait Time (between DB Calls)
• RT Unaccounted-for
• Improve process performance
• Find and analyze biggest time consumers
• Reduce amount of time spent, e.g.:
• CPU: SQL Tuning
• Wait: Resource contention

28
 References

• Note 39817.1 Interpreting Raw SQL_Trace

• Book to learn more about Event 10046 Trace
• Title: “Optimizing Oracle Performance”
• Year: 2003
• Author: Cary Millsap
• Publisher: O’Reilly

29