EAS Oracle Training

Performance Tuning

Amit Kumar
May 22nd 2007
 Explain Plan for this session
• The Problem
• Good performance habits
• The approach
• Define the problem
• Instance Tuning: Examine OS
• Instance Tuning: Examine Oracle Stats
• Instance Tuning: Implement and measure changes
• Using SQL Trace and TKPROF
• SQL Trace File Example
• Formatting Trace Files with TKPROF
• TKPROF File Example
• Understanding Explain Plan – Query Optimizer
• Optimizer Statistics Gathering
• Troubleshooting Network Performance
 The Problem

• Perception – Oracle Performance tuning is wizardy

– Why? – Too many things to check.. Too many ratios to figure
out..
– Ratios should be > 90%, some even > 99% else performance is
bad
– Any drop in ratios = performance loss
– Larger the Oracle caches = Better system performance
• Result – DBAs are now competing with one another to
throw more memory at various caches in Oracle SGA
 Good Performance Habits

• Seek out the bottlenecks on your system

• Cure the disease not just the symptoms
• Tuning should be part of the lifecycle of an application.
Most of the times tuning phase is left until the system
is in production. At this time, tuning becomes a reactive
fire-fighting exercise
• Tuning the resource hog on your system will provide you
maximum benefits
• Set tuning goals, stop tuning when you achieve them
 The Approach

• Define the problem

• Seek out the “OS Bottlenecks”
• Seek out the “Oracle Bottlenecks”
• Identify the real bottlenecks
• Make minimal changes as required
• Measure the effect of each change
 Define the problem

• Get the business expectations (SLA’s) for the

program/job
• Identify Scope of problem. Whole instance? One
program or specific operation?
• Identify time frame when problem occurs
• Quantify the slow down
• Identify any changes. Any OS change? Has more data
been loaded to system or the data volume has been
grown?
• If symptoms can be identified as local to a program,
focus on that program. Application tuning.
Instance Tuning: Examine OS: Detecting CPU
Bottlenecks

• CPU Bottlenecks (sar, top, prstat)

sar –o 5 5
essappu41:/apps/alice/aliceappl/fnd/11.5.0/resource> sar -o 5 5

SunOS essappu41 5.9 Generic_118558-20 sun4u 02/09/2007

06:44:47 %usr %sys %wio %idle

06:44:52 31 9 0 60

• If the CPU is used by a small number of high usage

programs then examine those programs
• If high usage programs are non-oracle, then determine
If they require that amount of cpu. If so, then can their
executions be delayed to off-peak hours
Instance Tuning: Examine OS: Detecting CPU
Bottlenecks

• High CPU Usage programs are oracle processes

• Once you identify high CPU usage oracle sessions, Drill
down at session level to examine what sql is running
and why is taking high cpu.
• Trace the session if needed
 Instance Tuning: Examine OS: Detecting I/O
Bottlenecks
• I/O bottlenecks (sar, iostat)
sar –d 5 1
essappu41:/apps/alice/aliceappl/fnd/11.5.0/resource> sar -d 5 1

SunOS essappu41 5.9 Generic_118558-20 sun4u 02/09/2007

06:48:23 device %busy avque r+w/s blks/s avwait avserv

06:48:28 md0 8 0.4 15 49 3.6 24.0

md1 0 0.0 0 0 0.0 0.0
md3 0 0.0 0 6 0.0 16.0

• i/o requests should be evenly balanced

• high disk queue numbers + high service times
à I/O contention
• Check the wait events in v$system_event to see if the top wait
events are i/o related
Instance Tuning: Examine OS: Detecting I/O
Bottlenecks

• i/o wait events are db file sequential read, db file

scattered read, db file single write and db file parallel
write. These are all events corresponding to i/os
performed against the data file headers, control files
or data files

• If any of these wait events correspond to high average

time, then investigate the problem

• Check v$sqlarea and sort sql’s by disk reads to find out

oracle sessions doing physical reads
 Instance Tuning: Examine OS: Detecting
Memory Bottlenecks
• vmstat –S 5 1

essappu41:/apps/alice/aliceappl/fnd/11.5.0/resource> vmstat -S 5 1
kthr memory page disk faults cpu
r b w swap free si so pi po fr de sr m0 m1 m3 m4 in sy cs us sy id
0 0 0 53270408 12000224 0 0 148 65 62 0 0 0 0 0 0 3353 7206 4838 27 5 67

• Do not go overboard on SGA sizing just to get high cache

hit ratios

• An example – One interface program run was performing

poor. To attain high library cache hit ratios, sga (shared
pool) was increased to 2gb. On examination, severe library
latch contention was found. Cause was lack of bind
variable usage. On fixing the application, system
performance came right back.
 Instance Tuning: Examine OS: Detecting
Network Bottlenecks
• Look at the network round trip ping times. (Ping, Tracert on Windows,
TraceRoute on Unix)
C:\Documents and Settings\kmra>tracert essdbpu51.emrsn.com
Tracing route to essdbpu51.emrsn.com [10.237.129.81] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 142.176.225.1
2 26 ms 26 ms 24 ms 10.140.47.193
3 27 ms 27 ms 26 ms 10.140.33.5
4 28 ms 28 ms 29 ms 152.162.104.121
5 240 ms 240 ms 240 ms 152.162.102.182
6 254 ms 254 ms 255 ms 172.21.250.169
7 264 ms 281 ms 290 ms 172.21.250.222
8 255 ms 254 ms 255 ms 10.237.133.194
9 256 ms 256 ms 257 ms essdbpu51.emrsn.com [10.237.129.81]
Trace complete.

• If the network is causing large delays in response time, then involve

network admin to find out possible causes
• Conversion programs need to be tested in similar network as
production
– Example – One conversion program was using db links to transfer data.
Program was tested on test servers located in same data center. When
this program ran in production, severe performance hit was found. On
investigation when we did ping from one production server to another,
there was large delay in response time as these two production servers
were located in different data centers.
 Instance Tuning: Examine Oracle: Detecting
Oracle Bottlenecks

• Oracle Statistics are examined and cross-referenced

with OS Stats
• Wait events means server process/thread has to wait
for an event to complete before further processing
• A server process can wait for
– A resource to become available (such as buffer or latch)
– An action to complete (such as an I/O)
– More work to do – idle events
• 220 Wait events in Oralce8i, about 400 in Oracle9i and
more then 800 in Oracle10g.
 Instance Tuning: Oracle Wait Interface

OWI Philosophy:
• The OWI is a performance tracking methodology that
focuses on process bottlenecks, better known as “wait
events”.
• OWI includes waits for I/O, locks, latches, background
processes, network latencies …
• OWI records and presents all the bottlenecks that a
process encounters from start to finish.
• Keeps tracks of the number of times and amount of
time a process spent on each bottleneck.
• Removing or even reducing the impact of major
bottleneck improves performance.
 Instance Tuning: Oracle Wait Interface

• OWI plays a significant role in determining the database

response time.

Response Time = Service Time + Wait Time

Service time is amount of time a process spends on a CPU.

Wait time is the amount of time a process wait for specific
resource to be available before continuing with processing.

• You can improve database response time by shortening

the service time, the wait time, or both.
 Instance Tuning: Oracle Wait Interface

• OWI Components:
A collection of a few dynamic performance views and
extended SQL trace file.
OWI provides statistics for wait events to tell how many
times and how long the session waited for an event to
complete.
• V$EVENT_NAME – not a dynamic view. Contains all the
wait events defined for your database.
• V$SYSTEM_EVENT – displays aggregated stats for all
wait events encountered by all oracle sessions since the
instance startup.
• V$SESSION_EVENT – contains aggregated wait event
stats by session for all sessions currently connected to
the instance.
• V$SESSION_WAIT – provides details info about the event
or resource that each session is waiting for
 Instance Tuning: Oracle Wait Interface

I/O Related Wait Events:

• Db file sequential read -> is initiated by SQL that perform single-block read
operations against indexes, undo segments. This is a single block read operation.
• db file scattered read -> is initiated by SQL that perform full scans operation
against tables and indexes. This is a multi-block read.
• Direct path read -> waits are driven by SQL that perform direct read operation
from the temporary or regular tablespaces.
• Direct path write -> waits are driven by SQL that perform direct write operation in
the temporary or regular tablespaces.
Locks related Wait Events:
• Latch free -> it means the process failed to obtain the latch in the willing-to-wait
mode after spinning _SPIN_COUNT times and went to sleep. (shared pool, Library
Cache, Cache Buffer Chains Latches)
• Enqueue -> Due to variety of enqueue types, this event can occur for many
different reasons. The common causes and appropriate action to take depends on
enqueue type and the mode that the sessions are competing for.
• Buffer busy waits -> occurs when a session wants to access a data block in the
buffer cache that is currently in use by other session.
 Instance Tuning: Examine Oracle

• Start with v$system_event to determine the top waits

on your system
Column names: Event, Total_waits, Total_timeouts,
Time_waited, Avg_wait
• Drill down to v$session_event
Same column names + SID. Event information for the session.
Stats are lost as session log off. Remember to get the SQL
executed by SID
• Get all the SQL for the session in question by joining
v$session_wait, v$session, v$sql
• Get the bottlenecking segment for this SQL by joining
v$session_wait, dba_data_files, dba_extents
 Instance Tuning: Examine Oracle

• Initialization Parameters that effect performance

– SGA_MAX_SIZE
– DB_CACHE_SIZE
– PGA_AGGREGATE_TARGET
– SHARED_POOL_SIZE
• Metalink Note 216205.1 “Database Initialization
Paraemters and Configurations for Oracle Apps 11i”.
• A poorly sized buffer cache results in excessive buffer
gets and physical I/O
• A poorly sezed shared pool results in library cache and
shared pool latch contention due to lack of space
Instance Tuning: Implement and Measure Changes

• Identify two or three changes that could potentially

improve the performance
• Implement one change at a time
• Measure affect of change against the baseline data
collected in performance definition phase
• It’s a iterative process. Know when to stop tuning.
 Using SQL Trace and TKPROF

• alter session set events '10046 trace name context forever,

level n';
– If the level is 4 or 12 then the trace file contains information about
the bind variables.
– If the level is 8 or 12 then the trace file contains information about
the events that the session waited for.
– Without any level it is equivalent to “alter session set
sql_trace=true”
• Some relevant init.ora parameters
user_dump_dest - Trace file location
timed_statistics=TRUE - Enables Timing
max_dump_file_size - Max size trace files (unlimited by default)
_trace_files_public=TRUE -changes permissions on trace files from
640 to 644
sql_trace=TRUE - enables trace globally
 Using SQL Trace and TKPROF
• How to find your trace file
– select spid from v$process a, v$session b where a.addr=b.paddr
and b.sid= (select distinct sid from v$mystat)

– ALTER SESSION SET TRACEFILE_IDENTIFIER= AMIT

select username, traceid from v$process

• How to Trace Someone Else’s Session

Exec sys.dbms_system.set_ev (sid, serial#, 10046, 12, ‘ ‘) - to turn on
Exec sys.dbms_system.set_ev (sid, serial#, 10046, 0, ‘ ‘) - to turn off

• Oracle Form level trace

Help -> Diagnostics -> Trace -> Trace with binds and waits

• Setting the Trace for MWA/telnet session or a particular user

– Log in as a System Administrator & move to Profile/System
– Check off the USER box - and enter username that you are tracing
– Search on the profile option - 'Initialization SQL Statement - Custom‘.
– Set following at the user level - with the following string -
BEGIN FND_CTL.FND_SESS_CTL('','', '', 'TRUE','','ALTER SESSION SET TRACEFILE_IDENTIFIER='||''''|| 'Amit' ||''''||'
EVENTS ='||''''||' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 '||''''); END;
 SQL Trace – What it contains

• Statement text for the cursors opened by the session

• Performance related statistics for these cursors
– Rows accessed by different stages during the execution of the
cursor (not the actual executions plan) which can be used later
by tkprof when doing Explain Plan to mesh different stages of
executions with rows accessed. This information is stored when
the cursor is closed. So if the trace file is closed before the
cursor is closed, this information will be missing
– Number of logical (current and consistent) and physical reads
– CPU and elapsed times (if timed_statistics are TRUE)
– parse, execute and fetch counts
– library cache misses
 SQL Trace File Example
Dump file /apps/betsys2/betsys2db/8.1.7/admin/betsys2/udump/betsys2_ora_461_DM_USER2.trc

*** TRACE DUMP CONTINUED FROM FILE ***

Oracle8i Enterprise Edition Release 8.1.7.4.0 - Production

With the Partitioning option
JServer Release 8.1.7.4.0 - Production
ORACLE_HOME = /apps/betsys2/betsys2db/8.1.7
System name: SunOS
Node name: essdbsu41
Release: 5.9
Version: Generic_112233-12
Machine: sun4u
Instance name: betsys2
Redo thread mounted by this instance: 1
Oracle process number: 74
Unix process pid: 461, image: oracle@essdbsu41 (TNS V1-V3)

*** SESSION ID:(41.34778) 2005-05-25 04:53:31.081

=====================
PARSING IN CURSOR #30 len=104 dep=2 uid=173 oct=42 lid=173 tim=30531412 hv=3741596375 ad='5de1c6c0'
ALTER SESSION SET TRACEFILE_IDENTIFIER='DM_USER2' EVENTS =' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 '
END OF STMT
EXEC #30:c=0,e=0,p=0,cr=0,cu=0,mis=0,r=0,dep=2,og=4,tim=30531412
=====================
PARSING IN CURSOR #21 len=194 dep=1 uid=173 oct=47 lid=173 tim=30531412 hv=1642868942 ad='5de1ce8c'
BEGIN FND_CTL.FND_SESS_CTL('','', '', 'TRUE','','ALTER SESSION SET TRACEFILE_IDENTIFIER='||''''||'DM_USER2' ||''''||'
EVENTS ='||''''||' 10046 TRACE NAME CONTEXT FOREVER, LEVEL 12 '||''''); END;
END OF STMT
EXEC #21:c=0,e=0,p=0,cr=0,cu=0,mis=0,r=1,dep=1,og=4,tim=30531412
WAIT #1: nam='SQL*Net message to client' ela= 0 p1=1952673792 p2=1 p3=0
=====================
 SQL Trace File Example
STAT #13 id=1 cnt=0 STAT#13 Cursor identifier
STAT #13 id=2 cnt=1 id=1 Row source id
STAT #13 id=3 cnt=1 cnt=1 Rows processed
STAT #13 id=4 cnt=1
STAT #13 id=5 cnt=1
=====================
PARSING IN CURSOR #26 len=538 dep=0 uid=30 oct=3
SELECT OT.NAME "Order Type", OT.ORDER_TYPE_ID, OT.DESCRIPTION DESCRIPTION, LU.MEANING PRIORITY FROM SO_ORDER_TYPES
OT, SO_LOOKUPS LU WHERE(LU.LOOKUP_TYPE(+) = 'SHIPMENT_PRIORITY' AND LU.LOOKUP_CODE(+) = OT.SHIPMENT_PRIORITY_CODE AND
SYSDATE BETWEEN NVL(OT.START_DATE_ACTIVE, SYSDATE) AND NVL(OT.END_DATE_ACTIVE, SYSDATE+1) AND (:A00 = 'N' OR
OT.ATTRIBUTE2 = 'Y') AND EXISTS ( SELECT 'X' FROM SO_HEADERS SH WHERE SH.ORDER_TYPE_ID = OT.ORDER_TYPE_ID AND
NVL(OPEN_FLAG, 'N') = 'Y') ) AND ( UPPER(OT.NAME) LIKE :KEYCOL) ORDER BY UPPER(NAME)
END OF STMT
PARSE #26:c=0,e=0,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=3
c=0 CPU time (in 10ms increments) r=0 Rows processed
e=0 Elapsed time (in 10ms increments) dep=0 Recursive depth of the statement
p=0 Physical reads 0 for user, 1 for stmt from PL/SQL
cr=0 consistent reads og=3 Optimizer Goal : 4-CBO, 3-RBO,
mis=0 library cache misses 2 = all, 1=first
WAIT #26: nam='client message' ela= 0 p1=1 p2=4726832 p3=0
WAIT #26: nam='client message' ela= 1 p1=1 p2=4726832 p3=0
BINDS #26:
bind 0: bfp=101b9e94 bln=04 vln=01 dty=1 mxl=04 scl=00 pre=00 flg=02
typ=chr value="N"
bind 1: bfp=101beb28 bln=2000 vln=09 dty=1 mxl=2000 scl=00 pre=00 flg=02
typ=chr value="INTL-SUB%"
typ=chr Bind Variable Type
value=="INTL-SUB%" Bind variable value
EXEC #26:c=1,e=0,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=3
WAIT #26: nam='db file scattered read' ela= 0 p1=5 p2=280 p3=5
WAIT #26: nam='db file scattered read' ela= 0 p1=5 p2=286 p3=3
WAIT #26: nam='db file scattered read' ela= 0 p1=5 p2=9139 p3=2
WAIT #26: nam='db file scattered read' ela= 1 p1=5 p2=9142 p3=2
nam=‘db file scattered read’ session wait event
ela= 1 elapsed time (in 10ms)
p1=5 p2=9142 p3=2 session wait parameters
WAIT #26: nam='db file scattered read' ela= 0 p1=5 p2=9146 p3=2
WAIT #26: nam='db file scattered read' ela= 0 p1=5 p2=9149 p3=8
WAIT #26: nam='db file scattered read' ela= 0 p1=5 p2=9158 p3=4
WAIT #26: nam='db file sequential read' ela= 0 p1=5 p2=9163 p3=1
WAIT #26: nam='db file sequential read' ela= 0 p1=5 p2=9165 p3=1
WAIT #26: nam='db file scattered read' ela= 0 p1=5 p2=280 p3=2
 SQL Trace File Example
=====================
PARSING IN CURSOR #1 len=125 dep=0 uid=173 oct=47 lid=173 tim=30531419 hv=651791082 ad='707b19dc'
declare dummy boolean;begin fnd_profile.get_specific(:name, :userid, :respid, :applid, :val, dummy,
:org_id,:server_id);end;
END OF STMT
PARSE #1:c=0,e=0,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=4,tim=30531419
WAIT #1: nam='SQL*Net message to client' ela= 0 p1=1952673792 p2=1 p3=0
WAIT #1: nam='SQL*Net message from client' ela= 1 p1=1952673792 p2=1 p3=0
BINDS #1:
bind 0: dty=1 mxl=2000(1998) mal=00 scl=00 pre=00 oacflg=01 oacfl2=0 size=2000 offset=0
bfp=01b2b9e0 bln=2000 avl=09 flg=05
value="MRP_TRACE"
bind 1: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=01 oacfl2=0 size=72 offset=0
bfp=01a932d0 bln=22 avl=04 flg=05
value=10159
bind 2: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=01 oacfl2=0 size=0 offset=24
bfp=01a932e8 bln=22 avl=04 flg=01
value=52348
bind 3: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=01 oacfl2=0 size=0 offset=48
bfp=01a93300 bln=22 avl=03 flg=01
value=401
bind 4: dty=1 mxl=2000(1998) mal=00 scl=00 pre=00 oacflg=01 oacfl2=0 size=2000 offset=0
bfp=01b2b1f0 bln=2000 avl=00 flg=05
bind 5: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=01 oacfl2=0 size=48 offset=0
bfp=01a93294 bln=22 avl=01 flg=05
value=0
bind 6: dty=2 mxl=22(22) mal=00 scl=00 pre=00 oacflg=01 oacfl2=0 size=0 offset=24
bfp=01a932ac bln=22 avl=01 flg=01
value=0
 Formatting Trace Files with TKPROF
• tkprof produces a formatted output from the trace files generated by SQL
trace or event 10046.
• The output generated by tkprof can be sorted by a variety of parameters to
separate the more expensive statements from the rest.
• The first part of the parameter specifies the stage
– prs - parse
– exe - execution
– fch - fetch
• The second part specifies the actual statistic
– mis - misses in library cache
– dsk - physical reads
– qry - consistent reads
– cu - current read
– cpu - CPU time
– ela - elapsed time

• Tkprof <trace file> <tkprof output file name> sort=exeela fchela waits=yes
explain=<user/passwd>
 TKPROF File Example 1
TKPROF: Release 9.2.0.6.0 - Production on Fri Oct 21 18:04:27 2005

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

Trace file: betsyt2_ora_10354.trc

Sort options: default

********************************************************************************
count = number of times OCI procedure was executed
cpu = cpu time in seconds executing
elapsed = elapsed time in seconds executing
disk = number of physical reads of buffers from disk
query = number of buffers gotten for consistent read
current = number of buffers gotten in current mode (usually for update)
rows = number of rows processed by the fetch or execute call
********************************************************************************

alter session set events='10046 trace name context forever, level 12'

call count cpu elapsed disk query current rows

------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 0 0.00 0.00 0 0 0 0
Execute 1 0.00 0.00 0 0 0 0
Fetch 0 0.00 0.00 0 0 0 0
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 1 0.00 0.00 0 0 0 0

Misses in library cache during parse: 0

Optimizer goal: CHOOSE
Parsing user id: 173

Elapsed times include waiting on following events:

Event waited on Times Max. Wait Total Waited
---------------------------------------- Waited ---------- ------------
SQL*Net message to client 1 0.00 0.00
SQL*Net message from client 1 0.00 0.00
********************************************************************************
 TKPROF File Example 2
TKPROF: Release 9.2.0.6.0 - Production on Tue Oct 25 21:28:00 2005

Copyright (c) 1982, 2002, Oracle Corporation. All rights reserved.

Trace file: betsy_ora_11152.trc

Sort options: exeela fchela
********************************************************************************
count = number of times OCI procedure was executed
cpu = cpu time in seconds executing
elapsed = elapsed time in seconds executing
disk = number of physical reads of buffers from disk
query = number of buffers gotten for consistent read
current = number of buffers gotten in current mode (usually for update)
rows = number of rows processed by the fetch or execute call
********************************************************************************

SELECT CPT.SERIAL_NUMBER PT_SL_NUMBER , DRA.SERIAL_NUMBER DR_SL_NUMBER ,

CPT.INVENTORY_ITEM_ID PT_ITEM_ID , DRA.INVENTORY_ITEM_ID DR_ITEM_ID
FROM
CSD_PRODUCT_TXNS_V CPT , CSD_REPAIRS DRA WHERE ACTION_TYPE IN('RMA',
'WALK_IN_RECEIPT') AND DRA.REPAIR_LINE_ID = :B1 AND CPT.REPAIR_LINE_ID =
DRA.REPAIR_LINE_ID AND NVL(CPT.SERIAL_NUMBER_CONTROL_CODE,1) > 1

call count cpu elapsed disk query current rows

------- ------ -------- ---------- ---------- ---------- ---------- ----------
Parse 1 0.06 0.04 0 0 0 0
Execute 1 0.42 0.41 0 0 0 0
Fetch 1 32.35 309.40 56563 205651 0 1
------- ------ -------- ---------- ---------- ---------- ---------- ----------
total 3 32.83 309.86 56563 205651 0 1

Misses in library cache during parse: 1

Optimizer goal: CHOOSE
Parsing user id: 173 (APPS) (recursive depth: 1)
 TKPROF File Example 2

Rows Row Source Operation

------- ---------------------------------------------------
1 NESTED LOOPS OUTER
1 NESTED LOOPS OUTER
1 HASH JOIN OUTER
1 NESTED LOOPS OUTER
1 HASH JOIN OUTER
1 HASH JOIN OUTER
1 NESTED LOOPS OUTER
1 HASH JOIN OUTER
1 FILTER
6303 HASH JOIN OUTER
6303 NESTED LOOPS OUTER
6303 NESTED LOOPS
6303 NESTED LOOPS
6303 NESTED LOOPS
6303 NESTED LOOPS
6303 HASH JOIN
43 TABLE ACCESS FULL CS_TXN_BILLING_TYPES
6303 HASH JOIN
37063 NESTED LOOPS
37063 NESTED LOOPS
1 TABLE ACCESS BY INDEX ROWID CSD_REPAIRS
1 INDEX UNIQUE SCAN CSD_REPAIRS_U1 (object id 82662)
37063 TABLE ACCESS BY INDEX ROWID MTL_SYSTEM_ITEMS_B
55932 INDEX RANGE SCAN MTL_SYSTEM_ITEMS_B_N9 (object id 200613)
37063 INDEX UNIQUE SCAN MTL_SYSTEM_ITEMS_TL_U1 (object id 38056)
10900 TABLE ACCESS FULL CS_ESTIMATE_DETAILS
6303 INDEX UNIQUE SCAN CS_TRANSACTION_TYPES_B_U1 (object id 32048)
6303 INDEX UNIQUE SCAN CS_TRANSACTION_TYPES_TL_U1 (object id 32341)
6303 INDEX UNIQUE SCAN MTL_UNITS_OF_MEASURE_TL_U2 (object id 38094)
6303 INDEX UNIQUE SCAN CSD_REPAIRS_U1 (object id 82662)
0 TABLE ACCESS BY INDEX ROWID OKC_K_HEADERS_B
0 INDEX UNIQUE SCAN OKC_K_HEADERS_B_U1 (object id 85477)
6102 TABLE ACCESS FULL CSD_PRODUCT_TRANSACTIONS
976289 TABLE ACCESS FULL CSI_ITEM_INSTANCES
1 INDEX UNIQUE SCAN OE_ORDER_HEADERS_U1 (object id 41952)
459001 TABLE ACCESS FULL HZ_PARTY_SITES
459001 TABLE ACCESS FULL HZ_PARTY_SITES
1 INDEX UNIQUE SCAN HZ_LOCATIONS_U1 (object id 172918)
353157 INDEX FULL SCAN OE_ORDER_LINES_U1 (object id 42102)
1 INDEX UNIQUE SCAN QP_LIST_HEADERS_TL_PK (object id 45951)
1 INDEX UNIQUE SCAN HZ_LOCATIONS_U1 (object id 172918)
 Understanding Explain Plan – Query Optimizer

• Query Optimizer determines the most efficient way to

execute a SQL statement.
• The output from the optimizer is a PLAN that describes
an optimum method of execution
• Optimizer Operations –
- Statement Transformation
- Choice of Optimizer Goal
- Choice of Access Path
- Choice of Join Orders
 Query Optimizer – Optimizer Operations
• Choice of Optimizer Goal (Optimizer_Mode)
• ALL_ROWS – Default. Best Throughput
• FIRST_ROWS_n – Best Response Time

• OPTIMIZER_FEATURES_ENABLE=10.0.0

• Controlling behavior of Optimizer

• CURSOR_SHARING
• DB_FILE_MULTIBLOCK_READ_COUNT
• OPTIMIZER_INDEX_CACHING
• OPTIMIZER_INDEX_COST_ADJ
• OPTIMIZER_MODE
• PGA_AGGREGATE_TARGET

• Choice of Access Path

• FULL TABLE SCAN
• ROWID SCAN
• INDEX SCAN
 Query Optimizer – Choice of Join Orders
• When two or more tables are joined together, Oracle must pull the columns from
the oracle table , creating a result set, which is a temporary of sorts.
• The oracle optimizer chooses one of the five methods to perform this join:
– a nested-loop join,
– a sort-merge join,
– a hash join,
– a cluster join,
– and ( new with Oracle 8i) an index join. The most common join are first three
listed here.

• A nested-loop join reads the first row from the driving table, or outer table, and
checks the other table, called the inner table, for the value. Nested-loops work
best when the driving table is rather small and a unique index is defined on the
inner table’s join table.
 Query Optimizer – Choice of Join Orders

• Hash Join are used for joining large data sets. A hash table on the smaller table
join key is built in memory. Larger table is then scanned probing the hash table
to find the joined rows. However, if hash table grows too big to fit into memory,
it is broken to into different partitions. As hash table partitions exceeds allocated
memory, parts are written to disk temp segments. After hash table creation,
larger table is scanned, broken into partitions written to disk temp segments.
Finally each hash table partition is read into memory and corresponding partition
for larger table is scanned probing the hash table partition to find the joined
rows. This process is repeated for all partitions.
 Query Optimizer – Choice of Join Orders

• A sort-merge join reads both of the joined tables by the join column and
then matches the output from these sorts. The big difference between
this join and the nested-loop is that in the sort-merge, no rows are
returned until after this matching process completes, where as the
nested-loop returns rows almost immediately. Sort-merge joins work
better with larger amounts of data or in situations where two joining
tables are roughly the same size and most of the rows are returned.

• Cluster joins are used instead of nested-loops when the join condition is
making reference to tables that are physically clustered together. The
CBO recognizes this condition and uses it when the tables being joined
are in the same cluster. Table are candidates for clustering when they
joined together and no other queries are typically run against one or the
other.
 Query Optimizer – Choice of Join Orders

• Index joins work on the concept that if all the information required for
the SQL statement is found in the index, the underlying table structure is
not accessed or referenced in the execution plan.

• Cartesian Join is used when there is no join condition between tables.

The Optimizer joins every row from one table to every row of other
table, creating the Cartesian product of two tables. This results in very
poor performance. Cartesian joins should be avoided by specifying a join
condition.
 Optimizer Statistics Gathering

• Do Not gather optimizer statistics on entire database

excessively such as nightly or weekly
– Execution Plans are not likely to change if data
distribution has not changed
• Do not gather statistics during peak intervals when
many users are accessing system as
– Gathering statistics invalidates cursor
– Gathering statistics require dictionary and object
level locks
• Statistics should be gathered selectively on tables
which are growing at rapid rate.
• Consider Automatic Statistics Gathering in 10G
 Optimizer Statistics Gathering

• Automatic Statistics Gathering in 10G

– Gather_Stats_Job – Scheduled Job by Default at db creation
– Calls Dbms_Stats.Gather_Database_Stats_Job_Proc which is
similar to Gather_Database_Stats procedure using GATHER
AUTO option
– Collects stats on objects when no previous stats OR existing
stats are stale (more than 10% rows updated)
– Stale Statistics are identified when STATISTICS_LEVEL init.ora
parameter is set to TYPICAL or ALL. Tables are monitored for
DML, truncation and information is updated in
User_Tab_Modifications

• System Statistics (CPU,I/O) in 10G

– Dbms_Stats.Gather_System_Stats
 Optimizer Statistics Gathering for Oracle
Applications Environment

• Use only FND_STATS or “Gather Schema Statistics” and “Gather Table

Statistics” concurrent program
– Do Not Use “Analyze table” or dbms_stats command directly
• Gathering Statistics Enhancements in 11i.10
– Gather Schema Statistics concurrent program has
• Auto Gather Option – This will gather statistics only on tables
which have changed. Utilizes the Table Monitoring feature.
• To enable schema monitoring,
– Use “exec fnd_stats.ENABLE_SCHEMA_MONITORING()”
– Or Set init.ora parameter statistics_level=typical (10g)
• Auto List Option – List only the objects which have changed
• It maintains history of statistics collection
• No Invalidate Option – Does not invalidates cursor
 Optimizer Statistics Gathering for Oracle
Applications Environment

• Should I gather statistics for SYS schema?

- On 8i, never gather stats for sys, because it is not supported.
- On 9i, gathering stats for sys is supported, but not necessary.
- On 10g, it is mandatory to collect stats for sys.

• How to collect statistics for SYS schema ?

For 9i:
execute dbms_stats.gather_schema_stats ('SYS',cascade=>TRUE,degree=><# cpus>);
For 10g:
Just run the script
$APPL_TOP/admin/adstats.sql

• Can table monitoring and STATISTICS_LEVEL = typical be used with 10g DB and
E-business suite?

– Yes, that is supported for both Apps 11i and R12.

You can run FND_STATS or the concurrent program "Gather Schema Statistics" with
the GATHER_AUTO option, to gather stats only on the tables have been changed.
 Troubleshooting Network Performance
• Network performance is often neglected
• Network with poor latency or low bandwidth impacts user response time
• A direct switch should be used between the application tier and data
server tier
– Latency should be ~1-2ms per round trip

• Use ping command to find out latency between db tier and apps tier. Use
packet size of 2k (not default 32 bytes) since this is default packet size for
Sqlnet Net traffic.
> ping -s oracledb 2048
PING oracledb: 2048 data bytes
2056 bytes from oracledb (10.237.129.90): icmp_seq=0. time=1. ms
2056 bytes from oracledb (10.237.129.90): icmp_seq=1. time=1. ms
2056 bytes from oracledb (10.237.129.90): icmp_seq=2. time=1. ms
2056 bytes from oracledb (10.237.129.90): icmp_seq=3. time=1. ms
2056 bytes from oracledb (10.237.129.90): icmp_seq=4. time=1. ms
2056 bytes from oracledb (10.237.129.90): icmp_seq=5. time=1. ms
----oracledb PING Statistics----
6 packets transmitted, 6 packets received, 0% packet loss
round-trip (ms) min/avg/max = 1/1/1
 Troubleshooting Network Performance
• Use ping command from client to find out latency between client and
apps tier. Vary the packet size to determine the average latency.
C:\Documents and Settings\amit_kumar>ping aptier -l 2048
Pinging aptier [10.237.129.11] with 2048 bytes of data:

Reply from 10.237.129.11: bytes=2048 time=15ms TTL=248

Reply from 10.237.129.11: bytes=2048 time=15ms TTL=248
Reply from 10.237.129.11: bytes=2048 time=15ms TTL=248
Reply from 10.237.129.11: bytes=2048 time=14ms TTL=248

Minimum = 14ms, Maximum = 15ms, Average = 14ms

C:\Documents and Settings\amit_kumar>ping aptier -l 512

Pinging aptier [10.237.129.11] with 512 bytes of data:

Reply from 10.237.129.11: bytes=512 time=13ms TTL=248

Reply from 10.237.129.11: bytes=512 time=13ms TTL=248
Reply from 10.237.129.11: bytes=512 time=13ms TTL=248
Reply from 10.237.129.11: bytes=512 time=13ms TTL=248

Minimum = 13ms, Maximum = 13ms, Average = 13ms

 Troubleshooting Network Performance
• Use Tracert (on Windows), TraceRoute (on Unix) utility
to determine number of hops and impact on latency
C:\Documents and Settings\kmra>tracert dbtier
Tracing route to dbtier [10.237.129.81] over a maximum of 30 hops:
1 <1 ms <1 ms <1 ms 142.176.225.1
2 26 ms 26 ms 24 ms 10.140.47.193
3 27 ms 27 ms 26 ms 10.140.33.5
4 28 ms 28 ms 29 ms 152.162.104.121
5 240 ms 240 ms 240 ms 152.162.102.182
6 254 ms 254 ms 255 ms 172.21.250.169
7 264 ms 281 ms 290 ms 172.21.250.222
8 255 ms 254 ms 255 ms 10.237.133.194
9 256 ms 256 ms 257 ms dbtier [10.237.129.81]
Trace complete.

• If the network is causing large delays in response time,

then involve network admin to find out possible causes
 Troubleshooting Network Performance
• Monitor network statistics including packet rates, error rates,
collision etc using tools such as netstat –I
> netstat -i
Name Mtu Net/Dest Address Ipkts Ierrs Opkts Oerrs Collis Queue
lo0 8232 loopback localhost 72060200 0 72060200 0 0 0
ce0 9000 q_storage-0 q_storage-0 173310429 0 91567714 0 0 0
ce1 1500 aptier-ce2-test aptier-ce2-test 2275146 0 716704 0 0 0
ce3 1500 q_data-3 q_data-3 2701647 0 249612814 0 0 0
ce5 1500 aptier-ce3-test aptier-ce3-test 2348877 0 636224 0 0 0
ce6 1500 aptiera aptiera 1542780 0 75042 0 0 0
ce8 9000 q_storage-8 q_storage-8 2361709 0 90204828 0 0 0
ce9 1500 q_data-9 q_data-9 448345966 0 240286920 0 0 0
 Troubleshooting Network Performance

• Oracle Application server web cache can be used to

reduce network traffic between the client and middle
tiers
– Compression. Caching of images, style sheets and
java script

• Refer metalink note Note:306653.1 – “Installing and

Configuring Web Cache 10.1.2 and Oracle E-Business
Suite 11i”
 Appendix: More Information

• Oracle Metalink Note 39817.1 “Interpreting Raw

SQL_TRACE ”
• Oracle Metalink Note 216205.1 “Database Initialization
Parameters for Oracle Applications 11i ”
• Oracle Metalink Note 169935.1 “Troubleshooting
Oracle Apps Performance issues”
Q&A
 Exercise to be done before next session

1.
– Enable user level trace with binds and waits for application user
“SYSADMIN”
– Log off, login again as SYSADMIN and submit “Active Users” report
– Disable trace and log off
– Identify trace file of “Active Users” report
– Read the trace file and find out bind variables being used
– Do a tkprof on trace and analyze the tkprof

• 2
– A end user complains that production oracle applications is running
very slow. List down your steps you will be performing to check
application and database performance
Thank You