!

"#$%&'()*"$+,#
-&./,$0&'()1$&,/"#2)13")0#20&'()
$'4)4&$('320&+)5"$+0&+#

6#"#.&$7)8&,03'
 About Jeremiah
! !"#$%&'()*+,(-)./!
! Working with Oracle since 1994
! Owner, ORA‐600 Consulting
" Architecture, scaling, performance
" Availability, stability, complex recovery
" Training, seminars, recruiting
! UW Certificate Program instructor
! Internals and nontrivial issue resolution
 Class objectives
! Learn to induce realistic application load in test
! Learn to create realistic failures and problems
! Learn to detect, assess and diagnose problems
! Learn appropriate pathways to resolution
! Apply critical thinking to emergent problems
! Learn to reduce outage times
! Learn self‐diagnostics and self‐resolution
! Student participation (hands on the keyboard)
! Have fun watching Oracle break
 Introductions
! Where you work
! Your role
! Background (optional)
! What you want to get from class
! A little about your environment
" Number of instances
" Criticality
" Team size
 Problem profiles
! Hangs ! Crashes
" Single‐sesson " Session/server/process
" Multi‐session " Whole instance
" Whole instance " ORA‐600, ORA‐7445
" Multi‐instance
! Curruption/data loss
! Spins " Files
" Server process " Blocks
" Background process " Logical
" Diabolical

7/16/2008 5
 Rationale
! Substitute for real‐world ordeals
! Hard to find good troubleshooters
! High cost of outages
! Opportunity for improvement
! Obscurity of diagnostic skills
" Not a standard DBA skill
" Not well documented
! Inadequacy of OWS first‐line
! Fun, exciting
 Most applicable environments
! Mission‐critical apps
" Heavily‐used public website
" Internal business‐critical systems
! Usefulness obviated by HA?
" (RAC, clusters, DataGuard, replication)
" Most issues NOT addressed by HA technologies
! Environments where DBA will never encounter 
such issues
" Allows professional development
 Section 1: Inducing Load
! Need a realistic load to induce hangs, etc.
! Resource contention is a problem of 
concurrency
! Under load, problems get worse
! Helps find scaling limits of a system
! An inactive site is no excuse for not learning
! Many recent options available
 Induced Load: Options
! Generated workload
" Artificially generated transactions
" Simulates a type of application (OLTP/DSS)
" Can be turned up to exhaust server resources
! Recorded workload
" 0%1,)#223+4#-+%&'()-,15)3%#6
" Tunable playback
" Less opportunity to ratchet up
 Load Testing Tools
! Application loading
" HP (Mercury) LoadRunner
" Borland (Segue) Silk Performer
" IBM Rational Robot
" Web Performance Suite
" OpenSTA (open source) 
" Swingbench (open source)
! Oracle sessions record/playback
" Oracle Database Replay
" Quest Benchmark Factory
" Hammerora (open source)
 Application Load Testing Tools
! Script (manually define) workload as a user
! Playback workload against application

! Advantages ! Shortcomings
" Can exercise many  " Probably will miss some 
services/databases database load and 
" Provides whole‐ application work
application loading " Many manual steps to 
" Scripted load can be  define and tune load
multiplied " Concurrent DB txns not 
coordinated (errors)

7/16/2008 11
 Oracle Session Record/Playback Tools
! Obtain workload from 10046 trace (or internally)
! Play back load faithfully or with wait time removed

! Advantages ! Shortcomings
" Real workload, not  " :#&'-);5)"13-+23+56<)%&3=)
7#1-8%,569 wait time can be 
" Captures all activity, not  removed
just one application " Only applies to DB, not 
" Some (DB Replay)  app or other services on 
coordinate txns in  which the app is 
proper order dependent

7/16/2008 12
 Swingbench
! Open‐source tool by Dominic Giles (Oracle UK)
! Synthetic load harness
! Useful canned workloads
" Order Entry
" Calling Circle
! Simple to roll your own workload
! Quick and easy to set up
! http://www.dominicgiles.com/swingbench
 Database Replay
! Part of 11! Real Application Testing
! Capture from earlier versions
" 9.2.0.8, 10.2.0.3, 10.2.0.4
! Allows workload to resemble real application
! Allows subsetting by user, app, etc.
! Premium option
! 9"&.$"&,:)13")+7$'(#)$22/"$'+#
 Swingbench step by step
! Assumes Oracle binaries already installed

1. Set up a dedicated test database
2. Add new service to tnsnames.ora on a client
3. Download and install Swingbench on client
4. Configure Swingbench for your environment
5. Create and populate the SOE schema
6. Run a load test
 Swingbench
1. Set up a dedicated test database
oracle@db02$ dbca -silent -createdatabase \
> -templatename General_Purpose.dbc -storagetype fs \
> -gdbname od08 -sid od08 -totalmemory 500 \
> -syspassword od08pw -systempassword od08pw \
> -emconfiguration none -characterset al32utf8 \
> -datafiledestination /opt/oracle/oradata \
> -initparams _disable_interface_checking=true
Copying database files...
Creating and starting Oracle instance...
Completing Database Creation...
Database creation complete.
For details check the logfiles at:
/opt/oracle/cfgtoollogs/dbca/od08.
Database Information:
Global Database Name:od08
System Identifier(SID):od08
 Swingbench
2. Add new service to client tnsnames.ora
! Become user oracle
jeremiah@db01$ su - oracle
password:

! Set environment
oracle@db01$ . oraenv
ORACLE_SID = [oracle] ? *
The Oracle base for ORACLE_HOME=/opt/oracle/app/oracle/product
/11.1.0/db_1 is /opt/oracle

! Add to $ORACLE_HOME/network/admin/tnsnames.ora:
od08=
(description=
(address=(protocol=tcp)(host=192.168.1.205)(port=1521))
(connect_data=(service_name=od08))
)
 Swingbench
3. Download and install on a client
jeremiah@db01$ wget http://www.dominicgiles.com/swingbench/swingbench22.zip
Length: 7504679 (7.2M) [application/zip]
Saving to: `swingbench22.zip'
100%[============================================>] 7,504,679 370K/s in 20s

jeremiah@db01$ unzip swingbench22.zip

creating: swingbench/
inflating: swingbench/.DS_Store
creating: swingbench/bin/
inflating: swingbench/bin/ccwizard
...
inflating: swingbench/winbin/sample/oeconfig.xml
inflating: swingbench/winbin/sample/soeconfig.xml
inflating: swingbench/winbin/sample/spconfig.xml
inflating: swingbench/winbin/swingbench.bat
inflating: swingbench/winbin/swingconfig.xml
 Swingbench
4. Configure Swingbench
! Read README.txt
! Modify ./swingbench.env:
export ORACLE_HOME=/opt/oracle/app/oracle/product/11.1.0/db_1
export JAVAHOME=$ORACLE_HOME/jdk/jre
export SWINGHOME=~/swingbench
export LOADGENUSER=jeremiah
export CLASSPATH=$JAVAHOME/lib/rt.jar:$JAVAHOME/lib/tools.jar:
$ORACLE_HOME/jdbc/lib/ojdbc5.jar:$SWINGHOME/lib/mytransactions.jar:
${SWINGHOME}/lib/swingbench.jar:$ANTHOME/ant.jar

! Modify ./bin/oewizard.xml:
<WizardConfig Name="Oracle Entry Install Wizard" Mode="LightsOut">
<Parameter Key="dbapassword" Value="od08pw"/>
<Parameter Key="connectionstring" Value="//192.168.1.205/od08"/>

! Modify ./bin/swingconfig.xml:
<ConnectString>//db02/od08</ConnectString>
<NumberOfUsers>30</NumberOfUsers>
<TransactionList WaitTillAllLogon="true" MinDelay="0" MaxDelay="100"
MaxTransactions="-1" QueryTimeout="60">
 Swingbench
5. Create and poplulate SOE schema
! Set environment:
jeremiah@db01$ . oraenv
ORACLE_SID = [oracle] ? *
The Oracle base for ORACLE_HOME=/opt/oracle/app/oracle/product
/11.1.0/db_1 is /opt/oracle

jeremiah@db01$ export DISPLAY=<displayhost>:0

! Create tablespaces:
jeremiah@db01$ rlwrap sqlplus sys@od08 as sysdba
SQL*Plus: Release 11.1.0.6.0 - Production

SQL> create tablespace soe

datafile '/opt/oracle/oradata/od08/soe.dbf' size 100M reuse
autoextend on next 50m maxsize unlimited
extent management local uniform size 100k nologging;
Tablespace created.

SQL> create tablespace soeindex

datafile '/opt/oracle/oradata/od08/soeindex.dbf' size 100M reuse
autoextend on next 50m maxsize unlimited
extent management local uniform size 100k nologging;
Tablespace created.
 Swingbench
5. Create and poplulate SOE schema (cont.)
! Run oewizard to set up sample schema and data:
jeremiah@db01$ ./oewizard
SwingBench Wizard
Author : Dominic Giles
Version : 2.2
Running in Lights Out Mode using config file : oewizard.xml
15:25:03 07/10 [DBUG] SQLScriptRunner
15:25:03 07/10 [DBUG] SQLScriptRunner
15:25:03 07/10 [DBUG] Step5
15:25:04 07/10 [DBUG] Step5
...
15:25:21 07/10 [DBUG] SQLScriptRunner
15:25:24 07/10 [DBUG] SQLScriptRunner
15:25:24 07/10 [DBUG] Step5
15:25:50 07/10 [DBUG] Step5
15:25:50 07/10 [DBUG] Step5
15:26:43 07/10 [DBUG] Step5
15:26:43 07/10 [DBUG] SQLScriptRunner
15:26:54 07/10 [DBUG] SQLScriptRunner
15:26:54 07/10 [DBUG] SQLScriptRunner
15:27:08 07/10 [DBUG] SQLScriptRunner
...
15:28:42 07/10 [DBUG] Step5
Starting script ../sql/soecreateuser.sql
Script completed in 0 secs
Connecting
Connected
Starting script ../sql/soepackage.sql
Script completed in 2 secs
Populating Customers
Populated Customers in 25 secs
Populating Orders
Populated Orders in 53 secs
Starting script ../sql/soeconstraints.sql
Script completed in 11 secs
Starting script ../sql/soeindexes.sql
Script completed in 13 secs
Exiting LightsOut Session
 Swingbench
6. Run a load test
jeremiah@db01$ ./charbench
Author : Dominic Giles
Version : 2.2

Results will be written to results.xml.

Hit Return to Start & Terminate Run...

Users : 15 TPM : 246 Nested TPM : 0

<return>

Users : 0 TPM : 234 Nested TPM : 0

Completed Run.
 Swingbench
Demo GUI
! Start remote terminal connection to PC in lab
! Start Xmanager X server on PC
! Set DISPLAY on swingbench host to lab PC
! Run swingbench

! GUI version allows:
" Start/Stop/Restart
" Parameter changes
" Runtime statistics
 DB Replay step by step
1. Prepare database for capture
2. Capture a workload
3. Clone DB to capture start SCN
" We will use Flashback Database
4. Move workload to clone DB
" Unnecessary for us; see above
5. Run workload
 DB Replay
1. Prepare database for capture
! Make sure flashback is on
jeremiah@db02$ sqlplus / as sysdba

SQL> shutdown immediate

SQL> startup mount
SQL> alter database archivelog;
SQL> alter database flashback on;
SQL> alter database open;

! Create a capture directory
SQL> create directory wloadcap as '/opt/oracle/wloadcap';

! Make sure there is some load to capture
jeremiah@db01$ ./charbench
 DB Replay
2. Capture a workload
! Start capture
jeremiah@db01$ sqlplus sys@od08 as sysdba

SQL> exec dbms_workload_capture.start_capture( -

name=>'od08cap', dir=>'WLOADCAP')
PL/SQL procedure successfully completed.

! >#+-)#)?8+35@)"5#&?8+35)A5-)-85)(-#,-)B:C
SQL> column name format a20
SQL> select name,
to_char(start_time,'yyyy-mm-dd hh24:mi:ss') start_time,
start_scn
from wrr$_captures;
NAME START_TIME START_SCN
-------------------- ------------------- ----------
od08cap 2008-07-10 17:04:49 894606

! End the capture
SQL> exec dbms_workload_capture.finish_capture
PL/SQL procedure successfully completed.
 DB Replay
3. Clone (flash back) to start SCN
! Normally you would clone from a backup:
RMAN> duplicate target database to od08clone until scn 894606;

! Flashed back DB is the clone for this demo
oracle@db02$ sqlplus / as sysdba
SQL> shutdown immediate
SQL> startup mount
ORACLE instance started.
SQL> flashback database to scn 894606;
Flashback complete.
SQL> alter database open resetlogs;
Database altered.
 DB Replay
5. Run workload
! Normally we move workload files to clone
oracle@db02$ ls /opt/oracle/wloadcap
oracle@db02$ scp -r /opt/oracle/wloadcap db03:/opt/oracle/

! Here we flashed back so leave them in place
jeremiah@db01$ rlwrap sqlplus sys@od08 as sysdba
SQL> exec dbms_workload_replay.process_capture( -
capture_dir=>'WLOADCAP')
PL/SQL procedure successfully completed.
SQL> exec dbms_workload_replay.initialize_replay( -
replay_name=>'od08cap', -
replay_dir=>'WLOADCAP')
PL/SQL procedure successfully completed.
SQL> exec dbms_workload_replay.prepare_replay( -
think_time_scale=>0)
PL/SQL procedure successfully completed.

! Also move replay files to workload clients
jeremiah@db01$ scp oracle@db02:/opt/oracle/wloadcap/* replay/
 DB Replay
5. Run workload (cont.)
! Launch the workload replay clients
jeremiah@db01$ wrc system/od08pw@od08 replaydir=replay \
> connection_override=true
Workload Replay Client: Release 11.1.0.6.0 - Production
Wait for the replay to start (21:06:32)

! Start the replay
jeremiah@db01$ rlwrap sqlplus sys@od08 as sysdba
SQL> exec dbms_workload_replay.start_replay
PL/SQL procedure successfully completed.

! Back in wrc:
Replay started (21:09:01)
Replay finished (22:29:53)
 Section 2: Inducing problems
! Hangs ! Crashes
" Single‐sesson " Session/server/process
" Multi‐session " Whole instance
" Whole instance " ORA‐600, ORA‐7445
" Multi‐instance
! Curruption/data loss
! Spins " Files
" Server process " Blocks
" Background process " Logical
" Diabolical

7/16/2008 30
 Hangs
! One or more sessions getting "stuck"
! Really means waiting on something
! Locks, latches, I/O, object serialization
! Hanging sessions may be holding resources 
needed by others
! Work ethic of waits
! Long (legitimate) waits vs. hangs
" Oracle's view
" Customer's view
 Single‐session hangs
! Simplest case: blocked by an enqueue
SESS1> select order_id from orders
where order_date > sysdate - .1 for update;
SESS2> update orders set order_date=sysdate
where order_id = 26104;
SESS3> column program format a15 trunc
SESS3> column event format a45
SESS3> select sid, program, event, state,
seconds_in_wait, blocking_session
from v$session
where event like 'enq%'
or sid in (select blocking_session
from v$session
where event like 'enq%');

! Worse when a blocked session is holding resources
! Save above script as locks.sql
! !&546%-5(@
 Single‐session hangs (cont.)
! Session hang on uninstrumented operation
SESS1> create table t as
select * from all_objects where 1 = 0;
SESS1> exec dbms_stats.gather_table_stats(user,'T')
SESS1> alter session set "_optimizer_search_limit" = 100;
SESS1> select sys_context('USERENV','SID') from dual;
SESS1> select * from
t t1,t t2,t t3,t t4,t t5,t t6,t t7,t t8,t t9,t t10,
t t11,t t12,t t13,t t14,t t15,t t16,t t17,t t18,t t19,t t20,
t t21,t t22,t t23,t t24,t t25,t t26,t t27,t t28,t t29,t t30,
t t31,t t32,t t33,t t34,t t35,t t36,t t37,t t38,t t39,t t40,
t t41,t t42,t t43,t t44,t t45,t t46,t t47,t t48,t t49,t t50,
t t51,t t52,t t53,t t54,t t55,t t56,t t57,t t58,t t59,t t60,
t t61,t t62,t t63,t t64,t t65,t t66,t t67,t t68,t t69,t t70,
t t71,t t72,t t73,t t74,t t75,t t76,t t77,t t78,t t79,t t80,
t t81,t t82,t t83,t t84,t t85,t t86,t t87,t t88,t t89,t t90,
t t91,t t92,t t93,t t94,t t95,t t96,t t97,t t98,t t99;

SQL> @waits
 Multi‐session hangs
! Worse case: blocked by an enqueue while 
holding resources needed by others
SESS1> select customer_id from orders where order_id =
(select max(order_id) from orders
where order_date > sysdate - .5
and mod(order_id,3) = 0) for update;
SESS2> update orders set order_date = sysdate
where order_date > sysdate - .5
and mod(order_id,3) = 0;
jeremiah@db01$ ./charbench
SESS3> @locks

! D2-+"+(-+4)E(F)25((+"+(-+4)3%4G+&A@
 Multi‐session hangs (cont.)
! Essential troubleshooting Q: who is at the head of 
the line and what are they doing?
! Another case: inter‐component chain of 
dependencies
SESS1> update customers set credit_limit = 1000
where cust_last_name = 'Edwards';
SESS2> alter session set ddl_lock_timeout = 10000;
SESS2> alter table customers modify
(cust_first_name null);
SESS3> column program format a15 trunc
SESS3> column event format a45
SESS3> select sid, program, event, state,
seconds_in_wait, blocking_session
from v$session where type != 'BACKGROUND';

! Save the above script as waits.sql
! Chains of resource holding
 Whole‐instance hangs
! Hang I/O calls by processes that can't time out
oracle@db01$ mkdir -p /opt/oracle/bct
root@db01# exportfs -i -o rw,no_root_squash /opt/oracle/bct
root@db02# mount -F nfs -o rw \
192.168.1.202:/opt/oracle/oradata/od08/bct /mnt/orabct
SYS> alter database enable block change tracking
using file '/mnt/orabct/bct.ora';
oracle@db02$ ./charbench
root@db02# /etc/init.d/nfs stop
SYS> @waits

! Most Oracle BG processes will time out/crash
" DBW"
" LGWR
" CTWR is an exception
 Spins
! Endless loops AKA "out to lunch"
! Process may be hanging or not
! Found with top or ps
! Consumes CPU resources
! If hanging may be holding resources needed 
by others
 Server process spins 
! Hang and spin in regexp bug
SQL> select 1 from dual where regexp_like(' ','^*[ ]*a');
oracle@db02$ ps -eo pid,pcpu,args | sort -n +1 | tail -10

! Also other sessions hang
! Total denial of service available to any user
 Server process spins
! Hang and spin on uninstrumented operation
jeremiah@db01$ sqlplus sys@od08 as sysdba
SQL> grant create any directory to soe;
SESS1> create directory mydir as '/tmp';
oracle@db02$ mknod /tmp/myfile p
SESS1> create table mytab (a number)
organization external (
type oracle_loader default directory mydir
access parameters (
records delimited by newline fields terminated by ','
(a)) location ('myfile'));
Table created.
SESS1> select * from mytab;
SQL> @waits
oracle@db02$ ps -eo pid,pcpu,args | sort -n +1 | tail -10
 Background process spins
! Spinning background procs can't always be 
killed safely
oracle@db02$ ps -eo pid,s,args | grep ora_arc
oracle@db02$ kill -STOP `ps -eo pid,args | grep ora_arc \
| grep -v grep | awk '{print $1}'`
oracle@db02$ ps -eo pid,s,args | grep ora_arc
SQL> select group#, sequence#, archived, status from v$log
order by sequence#;
SQL> alter system switch logfile;
SQL> alter system switch logfile;
SQL> alter system switch logfile;
oracle@db02 $ ps -eo pid,pcpu,args | sort -n +1 | tail -10
SQL> column event format a45
SQL> select event, state, seconds_in_wait from v$session
where type = 'BACKGROUND' and program like '%LGWR%';
 Crashes
! Usually ORA‐00600 and ORA‐07445
! Single process crash #$" take down whole instance
! ORA‐00600: internal error code, arguments: [] [] [] []
" First argument tells you where in the code
" Additional arguments provide more information
" Process/session does not always die
" Not necessarily an emergency (anecdotes)
" OERI
! ORA‐07445: exception encountered: core dump [] []
" Core dump
" First argument tells you where in the code (10!+)
" Second argument is the signal (kill ‐l)
" Additional arguments provide more information
 ORA‐00600 Example
! Simplest case in PL/SQL
SQL> declare
a exception;
pragma exception_init(a,-600);
begin
raise a;
end;

! Nicer, lets you specify the arguments
SQL> oradebug unit_test dbke_test dde_flow_kge_ora 12333 0 0
 Bug that raises ORA‐00600
! Bug 6073325: SELECT QUERY WITH CONNECT BY 
PRIOR FAILS WITH ORA‐00600 [KKQCBYDRV:1]

SQL> select 1 from sys.table_privileges tp, user_objects uo

where tp.grantee in
(select 1 from sys.dba_role_privs
connect by prior granted_role = grantee
start with grantee = 'scott');

! Raises ORA‐600, but we are sill connected
! Not all ‐600 errors are fatal (most are not)
! Just a unhandled exception ‐ no reason to panic
 Bug that raises ORA‐00600
! Bug 6310653: ORA‐600 [KKQCTMDCQ: QUERY BLOCK 
COULD NOT] ON INSERT

SQL> create table t_6310653

(oid number, nm varchar(128), snm varchar(128));
SQL> create table r_6310653
(cid number, oid number, sc number,
dst number, rw number);
SQL> insert into t_6310653 values (1, 'foo', 'foo');
SQL> commit;
SQL> insert into r_6310653 (cid, rw, oid, sc)
with ors as
(select oid, score(1) rs from t_6310653
where contains(nm, '"foo"', 1) > 0
order by rs desc, nm, oid)
select 13, rownum, ors.oid, ors.rs
from ors where rownum <= 100;
 Bug that raises ORA‐00600
! Bug 6618235: ORA‐00600: [KKQCBYDRV:1] ON 
V$ARCHIVED_LOG WITH LEFT OUTER JOIN

SQL> select hh24.hh24, nvl (sum(blocks*block_size),0)

from (select trunc(sysdate-7 + (rownum-1)/24, 'hh24') hh24
from dual
connect by trunc(sysdate-7+(rownum-1)/24,'hh24')<=sysdate) hh24
left outer join (
select thread#,sequence#,next_time,
max(blocks) blocks,max(block_size) block_size
from v$archived_log
where dest_id = 1
group by thread#,sequence#, next_time)
on hh24.hh24=trunc(next_time, 'hh24')
group by hh24.hh24
order by 1;
 ORA‐07445 Example
! Simplest case: send a signal
SQL> select spid from v$process p, v$session s
where p.addr = paddr
and sid = sys_context('USERENV','SID');
oracle@db02$ kill -SEGV 2513

! Use PL/SQL
SQL> declare
a exception;
pragma exception_init(a,-7445);
begin
raise a;
end;
 Real ORA‐07445 bug
! Bug 6244173: ORA‐07445 IN QEESTRAVERSEEXPR FOR 
HIERARCHICAL QUERY
SQL> create table t2(col1 varchar2(60));
SQL> create table t1(c1 varchar2(60),
c2 varchar2(1),
c3 varchar2(60),
c4 varchar2(60));
SQL> explain plan for
select 1 from t1 a, t2 b ,t1 c
where b.col1 = 'xxslc_department'
and a.c1 not between c.c3 and c.c4
start with a.c2='p'
connect by prior a.c1 between a.c3 and a.c4;

! Raises ORA‐HIIHJ)(%)?5)3%%G)+&)#35,-)3%A@
! Nature of a crashed process to generate a disconnect
! Continued use of dead connection gives app:
" ORA‐3114: Not connected to Oracle
" ORA‐1041: internal error. hostdef extension doesn't exist
! oerr ora 1041 ‐ Call support!
 Whole‐instance crashes
! Something causes a required background process 
to exit

! ORA‐600, ORA‐7445, I/O errors, etc.
" Can actually be any error that prevents the next step

! Some will restart,  some crash the instance

! Usually, but not always sensible
 11! Background Processes:
Which ones crash the instance?
9"3+#22) <#2+"&50&3' 9"3+#22) <#2+"&50&3'
;$.# ;$.#
=>?- =03.&+)+3'0"3,1&,# 03).#.3":)2#"@#" ??;K ?$'$(#$C&,&0:)?3'&03")9"3+#22)M
=A>n A#43),3()$"+7&@#"2 ??*; ?$'$(#$C&,&0:)?3'&03")9"3+#22
>6Bn 63C)2+7#4/,#")+33"4&'$03" 9G;I G'0#"+3''#+0),$0#'+:).#$2/"#.#'0
>D9E >7#+%53&'0 9?*; 9"3+#22).3'&03"
<nnn <&25$0+7#"2 9-9n 9"3+#22)25$F'#"2
<!A? A#23/"+#).$'$(#")5"3+#22 Bnnn B/#/#)+,#$'/5)5"3+#22#2
<!8n <$0$C$2#)F"&0#")5"3+#22#2 B?;> B/#/#)+33"4&'$03"
<G=H <&$('32&C&,&0: 5"3+#22)H AN>* <&20"&C/0#4)"#+3@#":)5"3+#22
<G=I <&$('32&C&,&0: +33"4&'$03" A?-n A=>).$'$(#.#'0)2#"@#"
J<!= J,$27C$+%)4$0$)$"+7&@#" 5"3+#22 AO8A A#+3@#":)F"&0#"
6''' 63C 2+7#4/,#")5"3+#22#2 -nnn -7$"#4 2#"@#"2
KI8A A#43),3( F"&0#" -?>* -5$+#).$'$(#.#'0)+33"4&'$03"
K?<n I,3C$,)#'L/#/# 2#"@&+#)4$#.3'2 -?*; -:20#.).3'&03")5"3+#22
K?*; I,3C$,)#'L/#/# 2#"@&+#).3'&03" ODE? O&"0/$,)%##5#")31)0&.#)5"3+#22
??=; ?#.3":).$'$(#" 8nnn -5$+# .$'$(#.#'0)5"3+#22#2
 Instance crashes
! Simple case: kill an essential background 
process (tail the alert log)
oracle@db02$ ps -eo pid,args | grep ora_ckpt | grep -v grep
oracle@db02$ kill -KILL <pid>

! Simple case: send a SIGSEGV or SIGBUS to an 
essential background process
oracle@db02$ ps -eo pid,args | grep ora_dbrm | grep -v grep
oracle@db02$ kill -SEGV <pid>

" Raises ORA‐07445
 Instance crashes
! Cause fatal errors in essential background 
processes
SQL> select pid, program, background from v$process
where background = 1;
SQL> oradebug setorapid 16
SQL> oradebug call kgeasnmierr 4455547624 18446744071472029760
18446744071562043788 2 1 1

! Couldn't find a good ORA‐600 for background 
processes
 Take a backup
! Mount some mass storage
root@db01# exportfs -i -o rw,no_root_squash 192.168.1.205:/mnt/usb/od08
root@db02# mount -F nfs -o vers=3 192.168.1.202:/mnt/usb/od08 \
/mnt/remote

! Move flash recovery area and increase to DB size
SQL> select sum(bytes)/1024/1024/1024 gb from dba_segments;
SQL> alter system set db_recovery_file_dest = '/mnt/remote' scope=both;
SQL> alter system set db_recovery_file_dest_size = 5g scope = both;

! Delete old logs
jeremiah@db01$ rlwrap rman target sys@od08
RMAN> delete archivelog all;

! Fix block change tracking
RMAN> backup incremental level 0 database;

! Take a backup
SQL> alter database enable block change tracking
 Instance disappears without a trace

! Most common on Windows
! Usually clusterware or OS services
! Difficult to diagnose
 Corruption
! Catch‐all term
! Physical
" File headers
" Data blocks
" Controlfiles, logfiles, other logs
! Logical
" Application tables
" Data dictionary, SYS
! Break while backup runs (about 30 minutes)
SQL> select message from v$session_longops
where message like 'RMAN%' order by start_time;
 File header corruption
! Oracle bugs
! OS/hardware bugs
! Array mirror scenario is scary
! Header contents
" File ID, tablespace ID, create and checkpoint data
" Read at checkpoint time; implications of delay
! We use dd with conv=notrunc to corrupt
" dd if=foo of=bar bs=512 oseek=512 count=1 conv=notrunc
 File header corruption
! Simple example: write wrong file into header
oracle@db02$ dd if=/opt/oracle/oradata/od08/soeindex.dbf \
of=/opt/oracle/oradata/od08/soe.dbf \
bs=8192 iseek=1 oseek=1 count=15 conv=notrunc
15+0 records in
15+0 records out
SQL> alter system checkpoint;

! Check the alert log
! Restore datafile (under 2 minutes)
RMAN> restore datafile '/opt/oracle/oradata/od08/soe.dbf';
RMAN> recover datafile '/opt/oracle/oradata/od08/soe.dbf';
SQL> alter database datafile
'/opt/oracle/oradata/od08/soe.dbf' online;
 Data block corruption
! Simple example: garbage into a block
! Find a block in a known table
SQL> select min(dbms_rowid.rowid_block_number(rowid))
from soe.customers;
SQL> select customer_id, cust_email from soe.customers
where dbms_rowid.rowid_block_number(rowid) = 12;
oracle@db02 $ dd if=/opt/oracle/oradata/od08/soe.dbf bs=8192 iseek=12 \
count=1 | strings | grep Sachin.Neeson@oracle.com
oracle@db02$ dd if=$ORACLE_HOME/bin/oracle \
of=/opt/oracle/oradata/od08/soe.dbf \
bs=8192 oseek=12 count=1 conv=notrunc
1+0 records in
1+0 records out
SQL> alter system checkpoint;

! Check the alert log ‐ no errors!
! Read the block
SQL> select customer_id, cust_email from soe.customers
where dbms_rowid.rowid_block_number(rowid) = 12;
SQL> alter system flush buffer_cache;
SQL> select customer_id, cust_email from soe.customers
where dbms_rowid.rowid_block_number(rowid) = 12;

! Restore data block (read again)
RMAN> blockrecover datafile '/opt/oracle/oradata/od08/od08/soe.dbf' block 12;
 Controlfile corruption
! Get controlfile locations
SQL> show parameter control_files

! Write garbage into one controlfile
oracle@db02$ dd if=$ORACLE_HOME/bin/oracle \
of=/opt/oracle/oradata/od08/control01.ctl \
bs=8192 oseek=1 count=30 conv=notrunc

! Checkpoint
SQL> alter system checkpoint;

! Check alert log
! No need to restore from backup; just use one 
of the others
 Controlfile corruption
! Get controlfile locations
SQL> show parameter control_files

! Write garbage into all controlfiles
oracle@db02$ dd if=$ORACLE_HOME/bin/oracle \
of=/opt/oracle/oradata/od08/control01.ctl \
bs=8192 oseek=1 count=30 conv=notrunc

! Repeat for each of 3 copies
! Checkpoint
SQL> alter system checkpoint;

! Check backups
oracle@db02$ ls -l /mnt/remote/OD08/backupset/2008_07_15
RMAN> restore controlfile from
'/mnt/remote/OD08/backupset/2008_07_15/<backuppiece>.bkp';
RMAN> recover database;
SQL> alter database open resetlogs;

! Take a new level 0 backup
 Logfile corruption
! Make a second member for each group
SQL> alter database add logfile member
'/opt/oracle/oradata/od08/redo01a.log' to group 1;

" (Repeat)
! Cycle logs and run some load
SQL> alter system switch logfile;
SQL> alter system switch logfile;
SQL> alter system switch logfile;
jeremiah@db01$ ./charbench

! Find the current logs
SQL> column member format a55
SQL> select l.group#, member, archived, l.status
from v$log l, v$logfile lf where l.group# = lf.group#
order by l.group#, member;
 Logfile corruption (cont.)
! Corrupt one of the current logs
$ dd if=$ORACLE_HOME/bin/oracle \
of=/opt/oracle/oradata/od08/redo01.log bs=512 oseek=1 \
count=100 conv=notrunc

! Crash  and restart the instance
SQL> shutdown abort
SQL> startup

! Check the alert log
! Can corrupt both during diagnostic and 
resolution exercises
 Other vulnerable files
! Archived redo logs
! Flashback logs
! Flashback archives
! Block change tracking file
! Backups
 Logical corruption
! Erroneously changed data
" Missing/incorrect predicate (where clause)
! Human error/application bug
! Oracle bug (wrong results)

! Many tools to resolve
" Flashback query
" Flashback transaction
" Flashback table
" Flashback database
" Log Miner
" Traditional point‐in‐time recovery
" Mini‐clone recovery
 Logical corruption
! User oops: missing where clause
SQL> set feedback off
SQL> update customers set cust_first_name = 'Nimrod'
where rownum < 1000;
SQL> commit;
SQL> set feedback on
SQL> alter table customers enable row movement;
SQL> flashback table customers to timestamp
to_date('2008-07-13 22:20:00','yyyy-mm-dd hh24:mi:ss');

! Quality resolution requires examining 
"versions between" to get exact SCN of 
change. Increase undo_retention to support.
 Logical corruption
! Some insidious examples
! Update sys.obj$
SQL> select dbms_flashback.get_system_change_number from dual;
SQL> update obj$ set name = 'FOO' where object_id < 500;
SQL> commit;
SQL> select * from cat;
SQL> shutdown abort
SQL> flashback database to scn 1549974;
SQL> alter database open resetlogs;

! Much of dictionary is cached and seldom read
! Could be weeks or months (expired backups) 
before discovery
 Section 3: Detect and resolve
! Detection
" Manual
" Monitoring (automated)

! Resolution
" Investigative
" Manual
" Automated
 Detect and resolve
single‐session hangs
! Simplest case: blocked by an enqueue
SESS1> select order_id from orders
where order_date > sysdate - .1 for update;
SESS2> update orders set order_date=sysdate
where order_id = 26104;
SESS3> column program format a15 trunc
SESS3> column event format a45
SESS3> @lock.sql

! Locks and other identifiable waits should be 
monitored and detected
! Business rules can dictate approach to resolution
" Blocker blocking over " seconds is killed
" Success depends on resiliency of application's 
connection pool
! ORA‐00028: your session has been killed
 A manifesto for DBA sanity and
statistics‐based monitoring
! Leverage history in AWR
" dba_hist_active_sess_history
! Monitoring should:
" examine current waits and statistics (all)
" determine if current values fall outside STDDEV for
! Same time yesterday
! Same time last week
! Same time last month
! Same time last year
! Exposes more real problems than static thresholds
! Space management example
" dba_hist_seg_stat
The paramount importance of ASH
! ASH/AWR are extra‐cost options
! Inappropriate, bad decision
" More peoples' systems will run poorly
" Those who make money decisions don't 
understand the importance of this data
! Time‐series of waits per session per SQL
! Extremely rich
" explore v$active_session_history
 SASH
! Kyle Hailey (Perfvision) write a v$ view‐based pretty‐
good ASH
! Easy to install/use:
oracle@db02$ wget \
http://www.perfvision.com/ftp/sashpack151107.sh
oracle@db02$ wget http://www.perfvision.com/sql/ash/ashstart.sql
oracle@db02$ vi sashpack151107.sh #edit for your env
SQL> create user sash identified by sash
default tablespace users temporary tablespace temp;
SQL> grant connect, resource, analyze any, create table,
create view, alter session, create sequence,
create database link, unlimited tablespace,
create public database link to sash;
oracle@db02$ sh sashpack151107.sh

! Provides most of ASH features
" SASH.SASH_% tables
 Detect and resolve
single‐session hangs (cont.)
! Session hang on uninstrumented operation
SESS1> alter session set "_optimizer_search_limit" = 100;
SESS1> select sys_context('USERENV','SID') from dual;
SESS1> select * from
t t1,t t2,t t3,t t4,t t5,t t6,t t7,t t8,t t9,t t10,
t t11,t t12,t t13,t t14,t t15,t t16,t t17,t t18,t t19,t t20,
t t21,t t22,t t23,t t24,t t25,t t26,t t27,t t28,t t29,t t30,
t t31,t t32,t t33,t t34,t t35,t t36,t t37,t t38,t t39,t t40,
t t41,t t42,t t43,t t44,t t45,t t46,t t47,t t48,t t49,t t50,
t t51,t t52,t t53,t t54,t t55,t t56,t t57,t t58,t t59,t t60,
t t61,t t62,t t63,t t64,t t65,t t66,t t67,t t68,t t69,t t70,
t t71,t t72,t t73,t t74,t t75,t t76,t t77,t t78,t t79,t t80,
t t81,t t82,t t83,t t84,t t85,t t86,t t87,t t88,t t89,t t90,
t t91,t t92,t t93,t t94,t t95,t t96,t t97,t t98,t t99;

SQL> @waits
 Important wait event concept
! Even if SECONDS_IN_WAIT is increasing, this 
value is only valid over 1 second if STATE = 
'WAITING'

! WAITED SHORT TIME with high 
SECONDS_IN_WAIT is '30)$)"#$,)F$&0

! If the session is hanging, we are stuck in 
uninstrumented code
 If stuck in uninstrumented code
! Is there ASH data?
SQL> select event, session_state, sum(time_waited) tw
from v$active_session_history where session_id = 123
and sample_time > sysdate - (.25/24)
group by event, session_state order by tw;

! Is the session spinning in CPU?
SQL> select spid from v$session s, v$process p
where s.paddr = p.addr
and s.sid = <SID of hanging session>;
oracle@db02$ ps -eo pid,pcpu,args | sort -n +1 | tail -10

! Obtain errorstack (multiple if spinning)
SQL> oradebug set ospid <PID>
SQL> oradebug dump errorstack 1
SQL> oradebug tracefile_name

! Examine errorstack
" Search Metalink
" https://metalink.oracle.com/metalink/plsql/ml2_documents.sh
owDocument?p_database_id=NOT&p_id=153788.1
! Open an SR
 Dumps: errorstack
! Call stack trace and process state dump
! Dump for a hanging process or on error
! Levels:
" 0: dump error buffer
" 1: 0+call stack
" 2: 1+processstate
" 3: 2+context area
Frontloading an Oracle Support SR
! Main time wasters in an SR are
" Back‐and‐forth with diagnostic requests
" Waiting for the next ocurrence
" Doing meaningless operations at OWS request
! Frontloading diagnostics
" Errorstacks for spins and hangs
" Alert log
" RDA or ADR package
 Generating an ADR package
! Already took several errorstacks but ADR did not see 
errors so no incident is registered.
! Trigger a fatal error 
" Make sure you tell Support which one was "fake"
oracle@db02$ kill -SEGV 21999

! Tail the alert log
! Package the incident for Support
oracle@db02$ adrci
adrci> show homes
adrci> set home diag/rdbms/od08
adrci> show incidents
adrci> ips create package incident 33801
adrci> ips generate package 1 in /opt/oracle
oracle@db02$ unzip -l <package>.zip

! Upload the zip file to support when opening SR
" Contains everything they need
 Detect and resolve
multi‐session hangs
! Blocked by an enqueue while holding resources 
needed by others
SESS1> select customer_id from orders where order_id =
(select max(order_id) from orders
where order_date > sysdate - 1
and mod(order_id,3) = 0) for update;
SESS2> update orders set order_date = sysdate
where order_date > sysdate - 1
and mod(order_id,3) = 0;
jeremiah@db01$ ./charbench
SESS3> @locks

! Same rules for automatic lock detection and 
adaptive monitoring apply
! In addition, specific diagnostics can be taken 
automatically in problem situations
 Reactive diagnostics
! Monitors should take appropriate diagnostics
" Lock pile‐up should automatically trigger locks 
script
! In this situation, we can issue the generic 
hanganalyze dump
" Covers a variety of issues of chain of resource 
posession
SQL> oradebug setmypid
SQL> oradebug hanganalyze 1
 Hanganalyze: Chain of resource custody

! Shows who is waiting for whom and on what
" Level 1‐2: dependency graph and hang chains
" Level 3: add process dumps of hanging sessions
" Level 4: add process dumps of blocking sessions
" Level 5: add process dumps of all sessions in chains
" Level 10: dump all processes in instance !

! Easy to read compared to systemstate
 Hanganalyze dump interpretation
State of nodes 
([nodenum]/sid/sess_srno/session/state/start/finish/[adjlist]/predecessor): 
[0]/1/1/0xa6f8b0/IGN/1/2//none 
[1]/2/1/0xa70230/IGN/3/4//none 
[3]/4/1/0xa71530/IGN/5/6//none 
[4]/5/1/0xa71eb0/IGN/7/8//none 
[5]/6/1/0xa72830/IGN/9/10//none 
[6]/7/1/0xa731b0/IGN/11/12//none 
[7]/8/1/0xa73b30/IGN/13/14//none 
[8]/9/1/0xa744b0/IGN_DMP/15/18/[130]/none 
[9]/10/1/0xa74e30/IGN/19/20//none 
[10]/11/4202/0xa757b0/IGN/21/22/[130]/none 
[11]/12/1196/0xa76130/NLEAF/23/28/[49]/none 
[12]/13/1/0xa76ab0/IGN/29/30/[130]/none 
[37]/38/37/0xa85830/NLEAF/73/76/[50]/46 
[46]/47/15/0xa8adb0/NLEAF/91/92/[37][50]/none

! Now take a look at our hanganalyze dump
 Detect and resolve
multi‐session hangs (cont.)
! Essential troubleshooting Q: who is at the 
head of the line and what are they doing?
! Inter‐component chain of dependencies
SESS1> update customers set credit_limit = 1000
where cust_last_name = 'Edwards';
SESS2> alter session set ddl_lock_timeout = 10000;
SESS2> alter table customers modify
(cust_first_name null);
SESS3> @waits

! Save the above script as waits.sql
! Chains of resource holding also visible in 
v$wait_chains ‐ the online hanganalyze view
 11! V$WAIT_CHAINS
! Like hanganalyze
! Enqueues act funny
select chain_id, sid, blocker_sid, in_wait,
wait_event_text, in_wait_secs
from v$wait_chains;

! Available for monitoring
! Column names are better in this view
! Cause some library cache contention
SQL> alter session set ddl_lock_timeout = 10000;
SQL> begin
loop
execute immediate 'alter table soe.orders modify (order_mode varchar2(12))';
execute immediate 'alter table soe.orders modify (order_mode varchar2(8))';
end loop;
end;
/
SQL> @waits

! K%,5@
 11! V$WAIT_CHAINS
! Library cache contention
SQL> select chain_id, count(*) ct from v$wait_chains
having count(*) > 1 group by chain_id order by ct;
SQL> select chain_id, sid, blocker_sid, wait_event_text
from v$wait_chains wc,
(select chain_id cid, count(*) ct
from v$wait_chains having count(*) > 1
group by chain_id order by ct) m
where m.cid = wc.chain_id order by chain_id;

! Here we see real chains
! LC pin/lock holder queries used to be poor
! Now we can focus on blockers at head of chain
 Whole‐instance hangs
! Hang CTWR again:
SQL> alter system set db_recovery_file_dest='/opt/oracle'
scope=both;
SQL> shutdown immediate
SQL> startup
SQL> alter database enable block change tracking
using file '/mnt/remote/bct.ora';
oracle@db02$ ./charbench
root@db01# service nfs stop
SQL> alter system checkpoint;
SQL> @waits

! L85(5)?#+-()35#6)1()-%):L>M@
 Chasing CTWR's problem
! Check CTWR's waits
SQL> select sid, program, event, state,
seconds_in_wait, blocking_session
from v$session where program like '%CTWR%'

! Shows waiting on I/O, look for cause
root@db02# lsof -p <CTWR pid>
root@db02# truss -p <CTWR pid>

! Could still report to Oracle as enhancement 
request
! Follow same errorstack procedure to file SR
 Emergency diagnostics
! Things to get if a restart is imminent
" create table sav_ash as select * from 
v$active_session_history;
" create table sav_hang as select * from v$wait_chains;
" hanganalyze level 3 (x3)
" errorstacks of any blocking/hanging processes
! If you can't log in
" > 10! has 'prelim' connection
" 9% and lower there is still a way
10! and 11!: When you can't log in
! Use the prelim connection
oracle@od08$ sqlplus /nolog
SQL> set _prelim on
SQL> connect / as sysdba
SQL> oradebug setmypid
SQL> oradebug direct_access enable trace
SQL> oradebug direct_access disable reply
SQL> oradebug direct_access set content_type = 'text/plain'
SQL> oradebug direct_access select * from x$ksdhng_chains
SQL> oradebug tracefile_name

! Any x$ view can be specified
! Get ASH and other data as neede
! Use v$fixed_view_definition
SQL> select view_definition from v$fixed_view_definition
where view_name = 'gv$active_session_history';
 9%: When you can't log in
! Use a symbolic debugger on an active process
! Attach and call ksudss (dump system state)
oracle@db02$ ps -eo pid,pcpu,args | sort -n +1 | tail -10
oracle@db02$ gdb $ORACLE_HOME/bin/oracle 11559
(gdb) call ksudss(10)
(gdb) detach
(gdb) q

! Process is stopped while attached
! Same available for other dumps with some 
research
 Detect and diagnose
server process spin
! Hang and spin in regexp bug
SQL> select 1 from dual where regexp_like(' ','^*[ ]*a');

! Monitoring should detect runaways with ps
" Escalating time with high CPU
oracle@db02$ ps -eo pid,pcpu,time,args

! Dump errorstack to find culprit SQL
oracle@db02$ ps -eo pid,pcpu,args | sort -n +1 | tail -10
SQL> oradebug setospid 13120
SQL> oradebug dump errorstack 1
SQL> oradebug tracefile_name

! Try call stack search
! Open SR, search metalink for issue
 Detect and diagnose:
server process spins
! Hang and spin on uninstrumented operation
jeremiah@db01$ sqlplus sys@od08 as sysdba
SQL> grant create any directory to soe;
SESS1> create directory mydir as '/tmp';
oracle@db02$ mknod /tmp/myfile p
SESS1> create table mytab (a number)
organization external (
type oracle_loader default directory mydir
access parameters (
records delimited by newline fields terminated by ','
(a)) location ('myfile'));
Table created.
SESS1> select * from mytab;
SQL> @waits
oracle@db02$ ps -eo pid,pcpu,args | sort -n +1 | tail -10

! Class exercise: detect and diagnose
 Detect and diagnose:
background process spins
oracle@db02$ ps -eo pid,s,args | grep ora_arc
oracle@db02$ kill -STOP `ps -eo pid,args | grep ora_arc \
| grep -v grep | awk '{print $1}'`
oracle@db02$ ps -eo pid,s,args | grep ora_arc
SQL> select group#, sequence#, archived, status from v$log
order by sequence#;
SQL> alter system switch logfile;
SQL> alter system switch logfile;
SQL> alter system switch logfile;
oracle@db02 $ ps -eo pid,pcpu,args | sort -n +1 | tail -10
SQL> column event format a45
SQL> select event, state, seconds_in_wait from v$session
where type = 'BACKGROUND' and program like '%LGWR%';
 Detect and diagnose:
bug that raises ORA‐00600
SQL> select 1 from sys.table_privileges tp, user_objects uo
where tp.grantee in
(select 1 from sys.dba_role_privs
connect by prior granted_role = grantee
start with grantee = 'scott');

! Detection for these fatal errors requires alert 
log and application log monitoring
! search metalink for kkqcbydrv:1
 Detect and diagnose: 
bug that raises ORA‐00600
(as sys)
SQL> create table t_6310653
(oid number, nm varchar(128), snm varchar(128));
SQL> create table r_6310653
(cid number, oid number, sc number,
dst number, rw number);
SQL> insert into t_6310653 values (1, 'foo', 'foo');
SQL> commit;
SQL> insert into r_6310653 (cid, rw, oid, sc)
with ors as
(select oid, score(1) rs from t_6310653
where contains(nm, '"foo"', 1) > 0
order by rs desc, nm, oid)
select 13, rownum, ors.oid, ors.rs
from ors where rownum <= 100;
 Detect and diagnose:
bug that raises ORA‐00600
SQL> select hh24.hh24, nvl (sum(blocks*block_size),0)
from (select trunc(sysdate-7 + (rownum-1)/24, 'hh24') hh24
from dual
connect by trunc(sysdate-7+(rownum-1)/24,'hh24')<=sysdate) hh24
left outer join (
select thread#,sequence#,next_time,
max(blocks) blocks,max(block_size) block_size
from v$archived_log
where dest_id = 1
group by thread#,sequence#, next_time)
on hh24.hh24=trunc(next_time, 'hh24')
group by hh24.hh24
order by 1;
 Detect and diagnose: ORA‐07445 bug
SQL> create table t2(col1 varchar2(60));
SQL> create table t1(c1 varchar2(60),
c2 varchar2(1),
c3 varchar2(60),
c4 varchar2(60));
SQL> explain plan for
select 1 from t1 a, t2 b ,t1 c
where b.col1 = 'xxslc_department'
and a.c1 not between c.c3 and c.c4
start with a.c2='p'
connect by prior a.c1 between a.c3 and a.c4;

! Raises ORA‐HIIHJ)(%)?5)3%%G)+&)#35,-)3%A@
! Search Metalink for qeesTraverseExpr
 Detect and diagnose:
Instance crashes
! Cause fatal errors in essential background 
processes
SQL> select pid, program, background from v$process
where background = 1;
SQL> oradebug setorapid 16
SQL> oradebug call kgeasnmierr 4455547624 18446744071472029760
18446744071562043788 2 1 1

! Alert log monitoring is essential to catch this
! Availability monitoring finds down instances
! Diagnose like any other errored process
! Higher support severity
 Detect and diagnose: 
Logical corruption
! More accurate restore that doesn't clobber 
newer txns
SQL> set feedback off
SQL> update customers set cust_first_name = 'Nimrod'
where rownum < 1000;
SQL> commit;
SQL> set feedback on
SQL> select versions_startscn, versions_endscn, versions_xid
from customers
versions between timestamp sysdate-(.25/24) and sysdate
where cust_first_name = 'Nimrod';
SQL> select undo_sql from flashback_transaction_query
where xid = '00090015000003A1'

! Detection: generally user‐detected
" Other ideas: triggers, resource limits
 Detect and diagnose:
logfile corruption
! Find the current logs
SQL> column member format a55
SQL> select l.group#, member, archived, l.status
from v$log l, v$logfile lf where l.group# = lf.group#
order by l.group#, member;

! Corrupt both of the current logs
$ dd if=$ORACLE_HOME/bin/oracle \
of=/opt/oracle/oradata/od08/redo01.log bs=512 oseek=1 \
count=100 conv=notrunc
$ dd if=$ORACLE_HOME/bin/oracle \
of=/opt/oracle/oradata/od08/redo01a.log bs=512 oseek=1 \
count=100 conv=notrunc

! Would be caught (or good log copies made) 
immediately by a standby
 Logfile corruption (cont.)
! Crash  and restart the instance
SQL> shutdown abort
SQL> startup

! Check the alert log
! Possible resolution paths
" Skip corruptions
" Open an inconsistent DB
" Data unloader
" Patch the logfiles
 Review objectives
! Learn to induce realistic application load in test
! Learn to create realistic failures and problems
! Learn to detect, assess and diagnose problems
! Learn appropriate pathways to resolution
! Apply critical thinking to emergent problems
! Learn to reduce outage times
! Learn self‐diagnostics and self‐resolution
! Student participation (hands on the keyboard)
! Have fun watching Oracle break
 Next steps
! Internals courses
! Exploration/discovery
! Integration with organizational standards
! Integration with drill days
Q&A