You are on page 1of 13

Top 3 Oracle DBA SQL Scripts

HASHJOIN Data Labs

Prepared By: Vitaliy Mogilevskiy


# Fri: December 18 2015

DOCID: top_three_sql_scripts_oracle_dba
top_three_sql_scripts_oracle_dba 12-18-2015

Table Of Contents
I. Why?
1. ASH Monitoring
2. LOCK Monitoring
3. SPACE Monitoring
II. Conclusion
III. Additional Resources

Page 2 of 13 HASHJOIN Corporation 2015


top_three_sql_scripts_oracle_dba 12-18-2015

Why?
Currently I have 68 SQL scripts that were battle tested on a very busy database serving a popular
internet site. However if you asked me what are the TOP 3 sql scripts that I consider most useful - I
would say its the scripts that allow me to Monitor ASH, LOCKS and SPACE. Because if you have all
three areas covered to the point that it only takes you a few seconds to answer questions such as:

What is eating up all our space in the last 15 minutes?


What happened to the 800GB of free space we added last month?
What was causing all the TX locks 3 hours ago?
What SQL caused race condition and where did it come from?
How many sessions were blocked on row contention this morning?
Why did the analytics report take 4 times longer to complete last night?
What was the database doing at 3am this morning that consumed almost all I/O bandwidth
on the storage?
Where did the flood of trac come from? Was it front-end cluster of nodes or back-end DW
jobs?

then your life as an Oracle DBA is much easier than a DBA next door!

Do I have your attention? If so, lets dive in and Ill show you the three scripts I use that do
exactly that!

Page 3 of 13 HASHJOIN Corporation 2015


top_three_sql_scripts_oracle_dba 12-18-2015

ASH Monitoring
h1.sql is a little gem that I reach for when I am asked to troubleshoot a performance problem that
was reported hours ago. For example lets say its 9:30am and I get a call from a dev saying that her
APEX web application is hanging - she suspects a locking issue. All I need to know at this point is
when the problem was first reported - armed with the start time (lets say 7:00am) I simply do this:

wget https://s3.amazonaws.com/mve-shared/sql/h1.sql
sqlplus / as sysdba
@h1 0700 0930 0 -1 -1

Whats happening here? There are 5 parameters:

1: start HHMI [0700 = 7:00am]


2: end HHMI [0930 = 9:30am]
3: days back [0 = today; or 7 = seven days back]
4: instance [1 = INST_ID=1, give -1 for all]
5: service_hash [1225919510 = dba_services.name_hash, give -1 for all]

Page 4 of 13 HASHJOIN Corporation 2015


top_three_sql_scripts_oracle_dba 12-18-2015

and heres what I get back:

h1.sql output sample

Can you spot the problem? Its SQL*Net message from dblink, there are no locks - its a simple
problem of a badly written distributed query that is waiting on remote DB. That was easy! One other
hidden benefit of this script is that it saves its output in a table under a RUN_ID (in this case
RUN_ID=81) allowing you to compare the output of two RUN_IDs and clearly spot the dierences in
values grouped by WAIT EVENT. This is extremely valuable especially when someone says - this
used to work last week!, you simply do this:

sqlplus / as sysdba
@h1 0700 0930 7 -1 -1

The 7 in the third parameter instructs the script to look 7 days back for the same time frame (7
9:30am). The output of the above report will have its own RUN_ID=82 (next in sequence) and you
can now compare the two using a special script h1d.sql like so:

@h1d 81 82

Page 5 of 13 HASHJOIN Corporation 2015


top_three_sql_scripts_oracle_dba 12-18-2015

We didnt need OEM or any GUI apps to get to the bottom of the problem - all because the
diagnostics data is already in AWR tables and is available to us directly from command line /
sqlplus. Years ago - wed have to sample v$session_wait to get similar diagnostics, in fact, I
wrote a complete monitoring system that utilized such technique. But now, Oracle built this into the
core engine in a form of Active Session History (ASH) that automatically samples this data every
second with practically no overhead to the database! That is an incredible level of instrumentation
available to us and it would be a shame not to utilize it beyond what the OEM reports are
capable of.

Page 6 of 13 HASHJOIN Corporation 2015


top_three_sql_scripts_oracle_dba 12-18-2015

LOCK Monitoring
My #2 most used script is one that can detect and drill down into Oracles Blocking LOCKs across
an entire Oracle RAC Cluster - locks.sql. Heres how to use it:

wget https://s3.amazonaws.com/mve-shared/locks.sql
sqlplus / as sysdba
@locks.sql

NOTE: if you dont have wget on your system try curl instead:

curl -o locks.sql https://s3.amazonaws.com/mve-shared/locks.sql

And heres an example of locks fully decoded across multiple instances:

1 -- In this example (see lines 45-49) we have a case of:


2 -- SID-3084 on INST_ID=4 is blocking two SIDs on INST_ID=6
3 --
4 -- NOTE: lines 12-38 is just an FYI list of all current locks
6 --
7
8 17:31:45 VMOGILEVSKIY@LPROD3=> @locks.sql
9
10 blocked objects from GV$LOCK and SYS.OBJ$
11
12 INST_ID SID LMODE MIN_BLOCKED BLOCKED_OBJ
13 ---------- ---------- ---------- ----------- ----------------------------
14 3 3961 3 0 APPUSER_OWNER.DBJOBREQUESTS
15 3 3866 3 0 APPUSER_OWNER.DBJOBREQUESTS
16 5 3887 3 0 APPUSER_OWNER.DBJOBREQUESTS
17 3 3484 3 0 APPUSER_OWNER.DBJOBREQUESTS
18 3 3161 3 0 APPUSER_OWNER.DBJOBREQUESTS
19 3 2998 3 0 APPUSER_OWNER.DBJOBREQUESTS
20 3 2979 3 0 APPUSER_OWNER.DBJOBREQUESTS
21 3 2752 3 1 APPUSER_OWNER.DBJOBREQUESTS
22 3 2618 3 0 APPUSER_OWNER.DBJOBREQUESTS
23 3 2610 3 0 APPUSER_OWNER.DBJOBREQUESTS
24 3 2456 3 0 APPUSER_OWNER.DBJOBREQUESTS

25 3 2368 3 0 APPUSER_OWNER.DBJOBREQUESTS
Page 7 of 13 HASHJOIN Corporation 2015
top_three_sql_scripts_oracle_dba 12-18-2015

25 3 2368 3 0 APPUSER_OWNER.DBJOBREQUESTS
26 3 2243 3 0 APPUSER_OWNER.DBJOBREQUESTS
27 3 2134 3 0 APPUSER_OWNER.DBJOBREQUESTS
28 3 2132 3 0 APPUSER_OWNER.DBJOBREQUESTS
29 6 3854 3 0 APPUSER_OWNER.DBJOBREQUESTS
30 6 3507 3 0 APPUSER_OWNER.DBJOBREQUESTS
31 6 3417 3 0 APPUSER_OWNER.DBJOBREQUESTS
32 6 3303 3 0 APPUSER_OWNER.DBJOBREQUESTS
33 6 3222 3 1 APPUSER_OWNER.DBJOBREQUESTS
34 6 3135 3 0 APPUSER_OWNER.DBJOBREQUESTS
35 6 2804 3 0 APPUSER_OWNER.DBJOBREQUESTS
36 6 2786 3 0 APPUSER_OWNER.DBJOBREQUESTS
37 4 3818 3 0 APPUSER_OWNER.DBJOBREQUESTS
38 4 2869 3 0 APPUSER_OWNER.DBJOBREQUESTS
39
40 25 rows selected.
41
42 Elapsed: 00:00:00.03
43 blocked sessions from GV$LOCK
44
45 INST_ID BLOCKER_SID INST_ID BLOCKED_SID MIN_BLOCKED REQUEST
46 ---------- ----------- ---------- ----------- ----------- ----------
47 4 3084 6 3135 0 6
48 4 3084 6 3485 0 6
49
50 2 rows selected.
51
52 Elapsed: 00:00:00.02
53 blocked session details from GV$SESSION and GV$SQLTEXT
54
55 Instance........ : 6
56 Sid ............ : 3135
57 Serial ......... : 30604
58 Username ....... : APP1USER_NAME
59 SQL Id ......... : null
60 Prev SQL Id .... : gm424t8fyx3w6
61 Displayed SQL Id : gm424t8fyx3w6
62 Client Info .... : null
63 Machine ........ : dbt4.dc1.mydomain.com
64 OSuser ......... : dbt
65 Process ........ : 1234

66 Action ......... : JDBC Thin Client


Page 8 of 13 HASHJOIN Corporation 2015
top_three_sql_scripts_oracle_dba 12-18-2015

66 Action ......... : JDBC Thin Client


67 SQL_TEXT
68 ----------------------------------------------------------------------
69 select this_.WorkRequestId as WorkRequ1_1_0_, this_.CreateTime a
70 s CreateTime1_0_, this_.Event_Type as Event3_1_0_, this_.Status
71 as Status1_0_, this_.UserId as UserId1_0_ from DBJOBREQUESTS thi
72 s_ where this_.WorkRequestId = :1 and this_.Status=:2 for upda
73 te
74
75 Instance........ : 6
76 Sid ............ : 3485
77 Serial ......... : 45149
78 Username ....... : APP1USER_NAME
79 SQL Id ......... : null
80 Prev SQL Id .... : gm424t8fyx3w6
81 Displayed SQL Id : gm424t8fyx3w6
82 Client Info .... : null
83 Machine ........ : dbt5.dc1.mydomain.com
84 OSuser ......... : dbt
85 Process ........ : 1234
86 Action ......... : JDBC Thin Client
87 SQL_TEXT
88 ----------------------------------------------------------------------
89 select this_.WorkRequestId as WorkRequ1_1_0_, this_.CreateTime a
90 s CreateTime1_0_, this_.Event_Type as Event3_1_0_, this_.Status
91 as Status1_0_, this_.UserId as UserId1_0_ from DBJOBREQUESTS thi
92 s_ where this_.WorkRequestId = :1 and this_.Status=:2 for upda
93 te
94
95 10 rows selected.
96
97 Elapsed: 00:00:09.33
98 blocker session details from GV$SESSION and GV$SQLTEXT
99 (current or previous SQL)
100 Instance........ : 4
101 Sid ............ : 3084
102 Serial ......... : 8911
103 Username ....... : APP1USER_NAME
104 SQL Id ......... : null
105 Prev SQL Id .... : 629vx81ykvhpp
106 Displayed SQL Id : 629vx81ykvhpp

107 Client Info .... : null


Page 9 of 13 HASHJOIN Corporation 2015
top_three_sql_scripts_oracle_dba 12-18-2015

107 Client Info .... : null


108 Machine ........ : dbt1.dc1.mydomain.com
109 OSuser ......... : dbt
110 Process ........ : 1234
111 Action ......... : JDBC Thin Client
112 SQL_TEXT
113 ----------------------------------------------------------------------
114 update DBT_LOCK set FINISHED=:1 , VERSION=:2 where USER_ID=:3
115 and VERSION=:4
116
117 2 rows selected.
118
119 Elapsed: 00:00:10.13

This script presented few performance challenges because a self-join query against gv$lock
joined with sys.obj$ to get a list of blocked objects is very expensive in a cluster environment, in
fact its expensive even in a single instance environment. We also have to join gv$session with a
result of self-join query against gv$lock in order to get the SQL_TEXT of the sessions doing
blocking and being blocked - thats extremely slow as well.

To solve the above performance challenges I created two tables and indexed them appropriately:

gv table mapping

Once that was done it was a simple matter of replacing GV$ Table name with COPY Table name on
the key joins and performance improved dramatically. In fact, the script ran so fast that I created a
custom event in my monitoring system and started to trap occurrences of these locks for historical
purposes so that when a developer came to our team and asked if there were any DB locks/blocks
3 hours ago we could simply review our alerts and answer that question with authority providing
exact details on the race condition that caused a block. This was much more helpful than the
generic alert email wed get from OEM stating that session 123 is blocking many sessions on
instances 1,4 and 5 for example.

Page 10 of 13 HASHJOIN Corporation 2015


top_three_sql_scripts_oracle_dba 12-18-2015

SPACE Monitoring
I dont buy into the space is cheap mantra because when we are dealing with SAN/FC storage -
its actually quite expensive especially if its FLASH based. If you can eectively find the root cause
of the vanishing space in an Oracle database you can save a good chunk of money for your
organization and make your own life much easier by having fewer requests for new LUNs from the
storage team.

Back in 2014 I was working for a company that operated a very large internet site and our space
consumption was at the rate of 800GB every month, we were adding 4x400GB LUNs to our RAC
cluster every two months. Then, one month we almost doubled our space consumption rate for no
apparent reason. I decided that it was time to find the root cause of the issue and developed a
series of scripts that helped us find the root cause in no time. Ill show you how it works so you can
adopt it in your workflow and DB monitoring.

The two scripts are:

1. segs2: Fast Extending Segments in the last X minutes for given TS


2. segs3: Fast Extending Segments Since Last Datafile Creation for given TS

These scripts use Oracle AWR, specifically dba_hist_seg_stat and


dba_hist_seg_stat_obj so its important that you have a recent AWR snapshot to work with -
you can easily create one using the following command:

exec dbms_workload_repository.create_snapshot();

Be careful - dont add a call to dbms_workload_repository.create_snapshot into the actual


scripts and then run it over and over again - doing so can put a strain on a busy database.

Page 11 of 13 HASHJOIN Corporation 2015


top_three_sql_scripts_oracle_dba 12-18-2015

Once you have a fresh AWR Snapshot created its very easy to use the scripts - simply login to the
Oracle DB in question via sqlplus and call the script:

For example to find which segments grew in the last 60 minutes in a tablespace DATA run
segs2.sql as follows:

wget https://s3.amazonaws.com/mve-shared/segs2.sql
sqlplus / as sysdba
@segs2.sql 60 DATA

or if you are interested in what segments grew in a tablespace DATA since the last datafile was
added run segs3.sql as follows:

wget https://s3.amazonaws.com/mve-shared/segs3.sql
sqlplus / as sysdba
@segs3.sql DATA

Page 12 of 13 HASHJOIN Corporation 2015


top_three_sql_scripts_oracle_dba 12-18-2015

Conclusion
I am hopeful that the TOP 3 Oracle DBA scripts I described above will make it into your own toolkit
or will give you an inspiration to write your own versions that tackle these three key areas of an
Oracle Database Monitoring: ASH, LOCKs and SPACE.

Additional Resources
If you liked this article and would like more like it, then please subscribe to my newsletter -
Confessions of an Oracle DBA where I share tips, scripts and tricks Ive learned during almost two
decades in the tech field as an Oracle DBA: http://www.dbatoolz.com/subscribe

End.

Page 13 of 13 HASHJOIN Corporation 2015