Do-It-Yourself Session

Monitoring with OTop

How to use a custom top-like
program to monitor Oracle
 Waiting, Waiting, Waiting
ƒ While driving - when you hit traffic

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 2

 Waiting, Waiting, Waiting
ƒ When you’re on line at the bank

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 3

 Waiting, Waiting, Waiting
ƒ When you’re coordinating a meeting in
different timezones

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 4

 Is Waiting Important?
ƒ When you’re at the doctor’s office
ƒ Whenever you have things planned back
to back
ƒ Cascading lateness
ƒ Little wastes of time between events
ƒ Timeliness requires organization, to use
up all those little pieces of time

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 5

 Waiting in Computing
ƒ Unix process context switching
ƒ Waiting for other processes
ƒ Time to switch from one to another
ƒ System/kernel time not available to user procs

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 6

 Waiting in Computing
ƒ Disk I/O Subsystem

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 7

 Waiting in Computing
ƒ Network Latency

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 8

 Computing : Dining
ƒ Disk I/O : Seating Capacity
ƒ Waiting to get a table can be frustrating
ƒ CPUs : Cooks
ƒ Waiting for your waitor, if they don’t manage their
tables well, slow!
ƒ Memory : Kitchen size
ƒ You need enough cooks to handle what
ƒ Processes/Threads : Waiters
ƒ Need to work in parallel, minmize context switches!

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 9

 Oracle Waiting
ƒ Organized into named events
ƒ Exposed via Data Dictionary views

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 10

 Oracle Wait Events: Disk I/O
ƒ db file sequential read
ƒ Single block read (undo, rlbk, indx)
ƒ Table access by rowid
ƒ db file scattered read
ƒ Blocks read & “scattered” as written to buffer
cache

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 11

 Oracle Wait Events: Sorting
ƒ direct path read
ƒ Also during parallel query + hash joins
ƒ direct path write
ƒ Create table as select
ƒ Parallel dml

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 12

 Oracle Wait Events: SGA
ƒ buffer busy waits (read by other session)
ƒ another session is accessing that block
ƒ free buffer waits
ƒ No free buffers in buffer cache
ƒ library cache pin
ƒ Compiling/parsing PL/SQL + Views

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 13

 Oracle Wait Events: Locks
ƒ latch free
ƒ Serialize access, but no waiting queue
ƒ Process can spin, and request latch again
ƒ Enqueue
ƒ Semaphore, serialize access to something

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 14

 Oracle Wait Events: Other
ƒ db file parallel read
ƒ db file parallel write
ƒ control file parallel write
ƒ library cache lock
ƒ Log buffer space
ƒ log file parallel write
ƒ log file sequential read
ƒ log file switch
Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 15
 Oracle Wait Events: Other
ƒ log file switch completion
ƒ log file sync
ƒ SQL*Net message from client
ƒ SQL*Net message to client
ƒ RAC events

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 16

 How Do We Watch Events?
ƒ ENTER: Oracle Wait Interface!
ƒ Oracle exposes db kernel memory
structures via data dictionary views
ƒ Allows us to use SQL to keep an eye on
things
ƒ Non-transactional, constantly changing
ƒ Even while we are querying!!

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 17

 OWI: Important Views
ƒ V$event_name
ƒ V$system_event
ƒ V$session_event
ƒ V$session_wait
ƒ V$session_wait_history (10g)
ƒ Others?

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 18

 V$event_name
SQL> desc v$event_name;
Name Null? Type
--------------------- -------- -----------
EVENT# NUMBER
EVENT_ID NUMBER
NAME VARCHAR2(64)
PARAMETER1 VARCHAR2(64)
PARAMETER2 VARCHAR2(64)
PARAMETER3 VARCHAR2(64)
WAIT_CLASS_ID NUMBER
WAIT_CLASS# NUMBER
WAIT_CLASS VARCHAR2(64)
Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 19
 V$event_name example
SQL> SELECT event#, name
2 FROM v$event_name WHERE name like 'db%';

EVENT# NAME
---------- ------------------------------
6 dbms_file_transfer I/O
115 db file sequential read
116 db file scattered read
117 db file single write
118 db file parallel write
119 db file parallel read

6 rows selected.

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 20

 V$system_event
SQL> desc v$system_event;
Name Null? Type
--------------------- -------- --------
EVENT VARCHAR2(64)
TOTAL_WAITS NUMBER
TOTAL_TIMEOUTS NUMBER
TIME_WAITED NUMBER
AVERAGE_WAIT NUMBER
TIME_WAITED_MICRO NUMBER
EVENT_ID NUMBER
WAIT_CLASS_ID NUMBER
WAIT_CLASS# NUMBER
WAIT_CLASS VARCHAR2(64)

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 21

 V$system_event example
SQL> SELECT event, average_wait
2 FROM v$system_event
3 WHERE event like 'db file%';

EVENT AVERAGE_WAIT
--------------------------------- ------------
db file sequential read .67
db file scattered read 1.12
db file single write .01
db file parallel write .44
db file parallel read 1.52

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 22

 V$session_event
SQL> desc v$session_event;
Name Null? Type
------------------- -------- -------
SID NUMBER
EVENT VARCHAR2(64)
TOTAL_WAITS NUMBER
TOTAL_TIMEOUTS NUMBER
TIME_WAITED NUMBER
AVERAGE_WAIT NUMBER
MAX_WAIT NUMBER
TIME_WAITED_MICRO NUMBER
EVENT_ID NUMBER
WAIT_CLASS_ID NUMBER
WAIT_CLASS# NUMBER
WAIT_CLASS VARCHAR2(64)

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 23

 V$session_event example
SQL> SELECT sid, event, average_wait FROM v$session_event
2 WHERE event like 'db file%';

SID EVENT AVERAGE_WAIT

----- ------------------------------ ------------
137 db file sequential read 2.42
143 db file sequential read 3.03
154 db file sequential read 1.64
154 db file scattered read .93
155 db file sequential read 2.26
156 db file sequential read .16
157 db file sequential read .15
157 db file scattered read 1.04
157 db file parallel read .01
160 db file parallel write .44

10 rows selected.

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 24

 V$session_wait
SQL> desc v$session_wait
Name Null? Type
--------------------- -------- ---------
SID NUMBER
SEQ# NUMBER
EVENT VARCHAR2(64)
P1TEXT VARCHAR2(64)
P1 NUMBER
P1RAW RAW(4)
P2TEXT VARCHAR2(64)
P2 NUMBER
P2RAW RAW(4)
P3TEXT VARCHAR2(64)
P3 NUMBER
P3RAW RAW(4)
WAIT_CLASS_ID NUMBER
WAIT_CLASS# NUMBER
WAIT_CLASS VARCHAR2(64)
WAIT_TIME NUMBER
SECONDS_IN_WAIT NUMBER
STATE VARCHAR2(19)

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 25

 V$session_wait example
SQL> insert into sean_test select * from sean_test;

SQL> SELECT event

2 FROM v$session_wait
3 WHERE sid = ‘140’;

EVENT
----------------------------

free buffer waits

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 26

 Manual Querying: Problems
ƒ Constantly changing
ƒ Don’t have historical data
ƒ 10g has V$session_wait_history
ƒ Other?

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 27

 Is There A Better Way?
ƒ Want a polling tool which is low-impact
ƒ Want to use via SSH login
ƒ Easy install, Curses library might be ideal
ƒ Do we have examples to model something
new?

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 28

 We All Use + Love TOP!

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 29

 Why We Like Top
ƒ At a glance view of heaviest processes
ƒ Memory Usage
ƒ Disk Usage
ƒ Load Average
ƒ CPU Busy/Idle
ƒ Add/Remove/Sort columns
ƒ Control the update interval
ƒ Color highlighting
Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 30
 Have You Heard of mtop?

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 31

 mtop – help screen

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 32

 mtop - variables

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 33

 mtop - recommendations

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 34

 Why mtop is popular
ƒ At a glance view of processes
ƒ Slow Query feedback
ƒ db uptime
ƒ Version, enabled features, add-ons
ƒ Various Sorting options
ƒ Locking, and related waiting activity
ƒ Sorting activity, and related waiting
ƒ OS information, load average etc
Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 35
 Why mtop is Popular
ƒ Explain a process (SQL in a session)
ƒ Kill a process (session)

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 36

 Inspiration. We need otop!
ƒ Session monitoring w/drill down
ƒ System event monitoring w/drill down
ƒ Various other info
ƒ Mem, disk, swap, load
ƒ SID, Hostname, etc

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 37

 otop – system event monitoring

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 38

 otop – system event monitoring
ƒ otop was using this query:
SELECT a.sid,a.event,a.time_waited,
a.time_waited / c.sum_time_waited * 100 pct_wait_time,
round((sysdate - b.logon_time) * 24) hours_connected
FROM v$session_event a, v$session b,
(select sid, sum(time_waited) sum_time_waited
FROM v$session_event
WHERE event not in (...)
HAVING sum(time_waited) > 0 group by sid) c
WHERE a.sid = b.sid
AND a.sid = c.sid AND a.time_waited > 0
AND (a.time_waited / c.sum_time_waited) < 1
ORDER BY pct_wait_time desc

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 39

 otop – session monitoring

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 40

 otop – session monitoring
ƒ Performing a sort
ƒ SELECT a, b, c FROM sean_test order by a;
ƒ otop was monitoring using this query:
ƒ SELECT sid,seq#,event,p1,p2,p3, wait_time
FROM v$session_wait_history
WHERE sid = ?
AND event not in (…)

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 41

 otop - help

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 42

 otop - help
ƒ otop option “h”
ƒ Show commands that interactively change
otop’s behavior

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 43

 otop - parameters

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 44

 otop - parameters
ƒ SELECT name, value
FROM v$parameter
WHERE isdefault = ‘FALSE’
ƒ otop option “p”

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 45

 otop - prerequisites
ƒ Perl (tested on 5.8)
ƒ Oracle (tested on 9.2,10.2)
ƒ DBI (tested on 1.51)
ƒ DBD::Oracle (tested on 1.17)
ƒ Curses (tested on 1.14)

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 46

 otop - installing
ƒ Install Oracle 10g (Win32 or Unix)
ƒ On Windows install ActiveState Perl
ƒ http://www.activestate.com/perl/
ƒ perl –MCPAN –e shell
ƒ CPAN> install DBI
ƒ CPAN> install DBD::Oracle
ƒ CPAN> install Curses::Forms
ƒ Download otop.tgz from here:
ƒ http://www.iheavy.com/otop/
Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 47
 otop - supported
ƒ Linux 2.2, 2.4, 2.6 (any version w/Perl)
ƒ Windows XP, 2k, NT (Activestate Perl)
ƒ Oracle 9.2,10.2

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 48

 otop - future
ƒ How about toggling session history?
ƒ How about toggling session trace?
ƒ How about killing a session?
ƒ How about thresholds for certain events?
ƒ How about color coding for easier viewing
at-a-glance?
ƒ How about sorting
ƒ top disk I/O, top CPU, top MEM activity

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 49

 otop - future
ƒ How about db recommendations?
ƒ SGA settings, redolog file sizes, backups
ƒ Missing indexes, too many indexes
ƒ Missing bind variables in queries
ƒ Patch/upgrade recommendations
ƒ Security checks

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 50

 OWI Bible
ƒ Oracle Wait Interface
ƒ By Richmond Shee,
Kirtikumar Deshpande and
K. Gopalakrishnan

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 51

 Rally the Troops!

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 52

 Please Contact Me
ƒ Interested in this project?
ƒ Collaboration is a great way to learn about
Oracle, and expand your skills.
ƒ OS, Data Dictionary, Perl programming,
and so on
ƒ shull@iheavy.com
ƒ 917.442.3939

Saturday, September 23, 2006 © 2006 by Sean Hull <shull@iheavy.com> 53