Orac

leLinu
x
T
e
s
t(OL
T
)Us
e
rGu
ide

TableofContents
1. Introduction........................................................................................................................2
2. HardwareRequirements.....................................................................................................2
3. PreInstallation...................................................................................................................3
3.1. OSConfiguration.........................................................................................................3
3.1.1. OSPackages.............................................................................................................3
3.1.2. SystemConfiguration..............................................................................................4
3.1.3. StorageConfiguration..............................................................................................5
NFS..................................................................................................................................5
OCFS2.............................................................................................................................6
ASMLIB..........................................................................................................................6
ASM................................................................................................................................7
FILESYSTEM ..............................................................................................................7
4. OLTInstallationandConfiguration...................................................................................7
4.1. SetupofOracleforOLT............................................................................................8
4.1.1. SetupofOracleSoftware(MainReleasesandPatchsets)...................................8
4.1.2. SetupofOraclePatches.........................................................................................10
4.1.3. SettinguptheORACLE_VERSIONandpatchesforthekittouse....................11
5. ExecutionofOLTTests...................................................................................................11
5.1. Schedulingtests.........................................................................................................11
5.2. MonitoringTests........................................................................................................12
5.3. LogsandTestResults................................................................................................12
5.4. Terminatingtests........................................................................................................12
5.5. Resultsreporting.......................................................................................................12
6. ExecutingRACdestructivetests......................................................................................13
6.1. Runningautomateddestructivetests.........................................................................13
6.2. Runningmanualdestructivetests..............................................................................13
6.3Runningmanualdestructivetestswithfailback.........................................................15
6.3. Resultsreporting........................................................................................................16
7. CleaninguptheOraclesoftwareinstallations..................................................................16
8.DebuggingOLTissues.......................................................................................................17
1

8. AppendixA.......................................................................................................................17
8.1. ConfiguringOLT......................................................................................................17
9. AppendixB........................................................................................................................18
10. AppendixCOLTUtilities............................................................................................18

1.Introduction
ThisdocumentdetailsthestepstosetupandexecuteOLTkit.
Thestepstosetupthekitandexecutethetestsonyoursysteminclude
.VerificationofHardwareRequirements
.PreInstallation
a.OSConfiguration
b.StorageConfiguration
..OLTInstallationandConfiguration
.ExecutionofOLTtestsSingleinstance,RAC
.ExecutionofOLTDestructivetests
.ResultsCollationforAuditing

2.HardwareRequirements
Thefollowingaretheminimumrequirements

ServerRequirements

2ServersormoresetupasaRACcluster

For11gR2,1SCANname(with1SCANIP)foraRACcluster.

Oneachserver,
3IPaddressesperRACnode

2publicIPaddressesforpublicandvipinterfaces(Note:Theipaddressofthe
vipinterfaceshouldbeinsamesubnetasthepublicinterface);and

OneprivateIPfortheinterconnect.

Atleast2networkportsforinterconnectandpublicnetworks

onededicatednetworkportforinterconnect

onededicatednetworkporttobeusedforpublicandvipnetworks

DisableNUMA viatheBIOSsettings(ifitisprovided)
ForvalidatingNAS,adedicatednetworkportisrequired.AnadditionalIPaddressisrequired
forconfiguringthisinterface.
ForvalidatingSAN,SingleormultipleFCcardsfortheSAN.

StorageRequirementsStoragesuchasNASora SANwhichcanbeconfiguredforasm/
asmliborocfs2.

ThefollowingarethestoragerequirementsfortheOLTkit
2

Location

Space

Purpose

Ownership

/opt/oracle/oltest

50GB

TestKitSoftware

oracle:oinstall

/home/oracle

15GB

Runtimeworkarea

oracle:oinstall

/tmp

1GB

Temporaryfiles

oracle:oinstall

Thefollowingarethe storagerequirementsfortheTestData:
Location
/oltstorage

Space
300GB

Purpose
TestDataforsingle

Ownership
oracle:oinstall

node
/oltstorage

110GB

ForRACdatabaseon

oracle:oinstall

NFS
/oltstorage(for

50GB

OCFS2)

ForRACdatabaseon
OCFS2

Forasmuse

50GB(perfailure

ForRACdatabaseon

the appropriate

group)

asm/asmlib

devicenames
Forasmlibuse
thelabelname

3.PreInstallation
3.1.OS Configuration
3.1.1.OSPackages
ThefollowingrpmsshouldbeinstalledforOLT
1.apache1.3.2971.15orhigherorhttpd
2.sysstat
3.bindutils
4.kernel<type>devel(OEL4,RHEL4,OEL5,RHEL5)
5.ncornetcat
6.perl
7.procmail
8.oraclevalidated

9.gdb
10.sudo

oracle:oinstall

N/A

3.1.2.SystemConfiguration
Network
1.Edit/etc/hoststoincludethefullyqualifiednamefirstfollowedbytheothernames.
2.Edit/etc/hoststoincludelocalhostentry
3.Setupyoursystemsuchthathostnamereturnsthefullyqualifiedname(withdomainname)
4.ForRAC,thefollowingsetupisrequired
1.Ensure that/etc/hosts contains allthe public,vip andprivate node names( with
andwithoutdomainnames)onallthenodes.
2.Ensurethatthetwonodeshavetheirpublicandprivateinterfacesonthesamesubnet.
3.Ensure thattheinterface namefor the publicIP isthe same onbothnodes.Ensure
thattheinterfacenamefortheprivateinterconnectisthesameonbothnodes.

4.EnsurethatVIPforanyofthenodesdoesnotmatchthefollowing
pattern:
172.16.x.x
172.20.x.x
192.168.x.x
10.x.x.x
TheseIPaddressesareassumedtobeprivateandmaynotbefitforbeingusedasaVIP
orpublic

5.MakesurethatVIPisnotpingable.Thevipshouldbeinthesamesubnetasthepublic
iponallthenodes

6.For11gRelease2,ensurethefollowinginregardtoSCANIP:
1.HostnamecorrespondingtoSCANIPhastobelessthen15characters
2.SCANIPhastobeonthesamesubnetaspublicIP.

3.HostnamecorrespondingtoSCANIPshouldbeconfiguredonthe
domainnameserver(DNS)forRoundRobinresolutiontothreeaddresses
(recommended)oratleastoneaddress.

4.SCANIP shouldnotbepingable

Environment
For10gor11gR1,
1.

Ensurethattimeissynchronizedonalltheracnodesusingntp.

For11gR2
2.Useclustertimesynchronizationprocess
i.e

.stopntpdservice
.mv/etc/ntp.confetc/ntp.conf.org

3.EnsureIptablesareturnedoffwhilerunningRACtests.
I.eServiceiptablesstop
4.EnsureSELinuxisdisabledfrombootcommandlineorSELinux
configfile(/etc/selinux/config).

3.1.3.StorageConfiguration
Thefollowingstoragetypescanbeusedforthetestdata.OLTrequirestobeconfiguredeachtimeanew
storagetypeistobevalidated.
1.NFS(singlenode+RACtests) usingOracle'sDNFSortraditionalNFS.
2.OCFS2(singlenode+RACtests)
3.asmlib(RACtests)
4.filesystemssuchasext2/3(singlenode)
Note:
1.refertotheHardwareRequirements>StoragerequirementsSection
2.rawandblockdeviceasocr/votedisksisnotsupportedin11gR2

NFS
FollowthestepsbelowtosetupyourNFSstorageforsinglenodeorRACforOLT.
1.Createadirectory/oltstorage(defaultlocation)
2.CreateapartitionontheNASfilerofrequiredsize.
3.MounttheNASstoragewiththefollowingoptions
Forkernelversion>=2.6.25,thedeprecatedmountoptionnointrisremoved.
Singlenoderw,bg,hard,tcp,nfsvers=3,timeo=600,rsize=32768,wsize=32768
RACrw,rsize=32768,wsize=32768,hard,bg,nfsvers=3,tcp,actimeo=0,timeo=600
Forkernelversion< 2.6.25
Singlenoderw,bg,hard,nointr,tcp,nfsvers=3,timeo=600,rsize=32768,wsize=32768
RACrw,rsize=32768,wsize=32768,hard,nointr,bg,nfsvers=3,tcp,actimeo=0,timeo=600

Note:IfOracledatabaseDirectNFSisused(11gR1upwards),mountoptionsarerequiredforocrand
votediskbutnotrequiredfordatabasefiles.

OCFS2
FollowthestepsbelowtosetupyourOCFS2storageforsinglenodeorRACforOLT.
Note:ForRHEL,downloadandsetupocfs2fromoss.oracle.com
1.

Createadirectory/oltstorage(defaultlocation)

2.

CreateanOCFS2partitionofrequiredsize.

3. MounttheOCFS2partitionon/oltstorage
Fordetailsonthesetupandmountoptionspleaserefertotheocfs2documentationonoss.oracle.com

ASMLIB
FollowthestepsbelowtosetupasmlibonyourmachineandlabelyourdisksforOLT.
Note:ForRHEL,download,installandsetupasmlibfrom
http://www.o
racle.com/t
echnology/t

ec
h/linux/

asmlib/ind

ex.html

1.Identifythedisksyouwanttouseforasm(defaultisexternalredundancy),letssay/dev/sdh1and
/dev/sdi1
2.Toconfigureasmperformthebelowsteps.Thisneedstobedoneonallnodesunlessspecified.
1.Zerooutthedisksyouplantouseforasmbydoingthefollowing(onlyonthemasternode)
$ddif=/dev/zeroof=/dev/<device>bs=<blocksize>count=<sizeofdisk/blocksize>
2.Edit/etc/sysconfig/oracleasm
ORACLEASM_ENABLED=true
ORACLEASM_UID=oracle
ORACLEASM_GID=oinstall

ORACLEASM_SCANBOOT=true
ORACLEASM_CLEARBOOT=true

3.Loadoracleasmdriver

4.Createalabelonthediskstobeusedbyasm(onlyonthemasternode).Fore.g:foralabel
nameofVOL1
/etc/init.d/oracleasmcreatediskVOL1/dev/sdh1
5.Scanthedisksusingoracleasm
6.Listthedisksyouhavechosentobeusedforasm

ASM
FollowthestepsbelowtosetupyourdisksforasmforOLT
1.

Identifythedisksyouwanttouseforasm(defaultisexternalredundancy),letssay/dev/sdh.

2.

Zerooutthedisksyouplantouseforasmbydoingthefollowing(onlyonthemasternode)
1.

$ddif=/dev/zeroof=/dev/<device>bs=<blocksize>count=<sizeofdisk/blocksize>

FILESYSTEM
FollowthestepsbelowtosetupyourfilesystemforsinglenodeforOLT
1.

Createadirectory/oltstorage(defaultlocation)

2.

Createapartitionofrequiredsize.

3.

Mountthepartitionon/oltstorage

4.OLTInstallationandConfiguration
OLTisavailableintwoformats:
1.

Singlearchivedfile(usedforinstallationandexecution)

2.Setofrpms(hostedpackagesonly)
Singlearchivedfile:InstallationandConfiguration

Downloadthearchivefile(OLT<version>.tar.gz)fromoss.oracle.com/projects/oltto
<Location>

cd<Location>
7

tarxzvfOLT<version>.tar.gz

Editoltsetup.envasperyourrequirementsformode(SI/RAC),filesystem,
DNFS_CLIENT,hostname,storage(forasm/ocr/voteredundancyanddisknames),
user,racnoderelatedinformation(no.,nodenames,interconnectdetails,vip).More
detailsavailableinAppendixA.

ExecuteoltInstallerasroot

Note:
1.Ifthereisanychangeinconfigurationmentionedabove,/opt/oracle/oltest/oltschedule/oltconfigure
needstoberunmanuallyforreconfiguration.(RefertoAppendixA)
2.OLTissetupinthecontextof"oracle"user.Thetestscanberuninthecontextofanyuser.

4.1.SetupofOracleforOLT
DownloadandsetuptheOraclesoftwareforOLTasmentionedbelow.WhenOLTisrun,Oraclewillbe
installedbythefirsttestofOLT(installsilentracforRACandinstallsilentoracle<version>forsingle
node).

4.1.1.SetupofOracle Software(MainReleasesandPatchsets)
ForOracleDatabase10g/11gRelease:
1. Download
1. For10g,
1.DownloadOracle10.2.0.1CDs(bothDatabaseandClusterware)fromotn.oracle.com
basedonthearchitecture.
1.F
orx8664(
http://download.o
racle.co
m/otn/l

inux/o
racle10g/10201/10201_datab
ase_linux_x86

_
6
4.cpio.gz

fordatabase,
http://download.o
racle.co
m/otn/l

inux/o
racle10g/10201/10201_clusterware
_linux_x8
6_64.cpio.gz

forclusterware)
2.F
orx86.(
http://download.o
racle.co
m/otn/l

inux/o
racle10g/10201/10201_datab
ase_linux32.

zip

fordatabase,
http://download.o
racle.co
m/otn/l

inux/o
racle10g/10201/10201_clusterware
_linux32.z
ipforclusterware)
2.Ifusing10.2.0.2,Download10.2.0.2patchset(4547817)frommetalink.oracle.com
3.Ifusing10.2.0.3,Download10.2.0.3patchset(5337014)frommetalink.oracle.com
4.Ifusing10.2.0.4,Download10.2.0.4patchset(6810189)frommetalink.oracle.com
8


1. For11g,
1.For11g,OracleDatabase11gCDs(bothDatabaseandClusterware)fromotn.oracle.com
om Forx86_64,

(http://download.o
racle.co
m/otn/l

inux/o
racle11g/linux.x64_11gR
1_database.zip
fordatabase,
http://download.oracle.co
m/otn/l

inux/o
racle11g/linux.x64_11gR
1_clusterware
.zi
pforclusterware)
2.Forx86,
(http://download.oracle.com/otn/linux/oracle11g/linux_x86_11gR1_database.z
ipfordatabase,
http://download.o
racle.co
m/otn/l

inux/o
racle11g/linux_x86_11gR
1_clusterware
.
zip
forclusterware)
2.Ifusing11.1.0.7,Download11.1.0.7patchset(6890831)frommetalink.oracle.com
3.

For11gRelease2,
1.Forx86_64
http://download.o
racle.co
m/otn/l

inux/o
racle11g/R2/linux.x64_11gR
2_database_1o
f2.
zip

http://download.o
racle.co
m/otn/l

inux/o
racle11g/R2/linux.x64_11gR
2_database_2o
f2.
zip

http://download.o
racle.co
m/otn/l

inux/o
racle11g/R2/linux.x64_11gR
2_g
rid.zip)
2.For
x86(
http://download.o
racle.co
m/otn/l

inux/o
racle11g/R2/linux_11gR
2_database_1o
f2z
.ip

http://download.o
racle.co
m/otn/l

inux/o
racle11g/R2/linux_11gR
2_database_2o
f2z
.ip

http://download.o
racle.co
m/otn/l

inux/o
racle11g/R2/linux_11gR
2_g
rid.zip)

4.For11.2.0.2
http://www.o
racle.com/t
echn
etw
o
rk/d
atab
ase/ent
erp
ris
e
edition/downlo

ads/ind

ex.html

Thelinkprovidesinformationfordownloading11.2.0.2.
1.Forx86_64
https://updates.oracle.com/download/10098816.html
p10098816_112020_Linuxx8664_1of7.zip
and
p10098816_112020_Linuxx8664_2of7.zipfordatabase
p10098816_112020_Linuxx8664_3of7.zipforCRS
9

2.Forx86,
https://updates.oracle.com/download/10098816.html
p10098816_112020_LINUX_1of7.zip
and
p10098816_112020_LINUX_2of7.zipfordatabase
p10098816_112020_LINUX_3of7.zipforCRS

5.For11.2.0.3download,
1.Forx86_64,
https://updates.oracle.com/download/10404530.html
p10404530_112030_Linuxx8664_1of7.zip
and
p10404530_112030_Linuxx8664_2of7.zipfordatabase
p10404530_112030_Linuxx8664_3of7.zipforCRS

2.Forx86,
https://updates.oracle.com/download/10404530.html
p10404530_112030_LINUX_1of7.zip
and
p10404530_112030_LINUX_2of7.zipfordatabase
p10404530_112030_LINUX_3of7.zipforCRS

Executethefollowingforeachoftheabovementioneddownloadedfilestosetup
OraclefortheOLTkit

$/opt/oracle/oltest/oltschedule/utils/oltisocopy

4.1.2.SetupofOracle Patches
1.DownloadtherequiredsetofOraclePatches(PleaserefertoAppendixBbelow)to
/home/oracle/oraclepatch.DownloadallClusterware(CRS)patchesto/home/oracle/crspatch
2.RunthecommandbelowtosetuptheOraclepatchesfortheOLTkitfromthepatchdirectory
(mentionedinStep1)foreachdownloaddirectorylocation.
$/opt/oracle/oltest/oltschedule/utils/oltpatchcopy
10

3.Ifsettingupthepatchesmanually,thenrefertoAppendixB.
4.Therearetwopatchlevels,namelycriticalandextra.Thepatchescopiedunderthe
criticaldirectorywillbeappliedbydefaultbythekit.

4.1.3.Settingupthe ORACLE_VERSIONandpatchesforthe kitto

use
TheORACLE_VERSIONissetto11.2.0.3bydefaultin/opt/oracle/oltest/oltschedule/env/<nodename>/
oltschedule.env.If youareusingandinstallingoracleversionotherthan11.2.0.3,thenresetthe
ORACLE_VERSION.Tochangethisversionyoucaneditthisenvfileforallthenodes.
e.g.:In /opt/oracle/oltest/oltschedule/env/<nodename>/oltschedule.env.
For10.2.0.3:resetORACLE_VERSION to10.2.0.3
i.e.exportORACLE_VERSION=10.2.0.3
Forpatches,
Applyingcr
iti
ca
lpatch
es
Allthepatchesfoundunderthecriticalpatchesdirectorywillbeappliedbydefault.
Applyingext
rapatch
es(custom

erspec
ificrequi
re
d
patch
es)
Toapplytheadditionalpatchesplacedundertheextrapatchesdirectory,youwouldneedtoedit
thetestcaseenvfilesandaddPATCHLEVEL=2
e.g.Forsinglenode10.2.0.1editoltschedule/testcases/<hostname>/installsilent
oracle10201.env
TEST_PARAMS="PRODUCT_NAME=oracle:PRODUCT_VERSION=10.2.0.1:UNIX_GROUP
_NAME=oinstall:ORACLE_HOME_NAME=ohome_ora_10201:PATCHLEVEL=2"
Oncetheabovestepshavebeencompleted,OraclewillbeinstalledbyOLTwhenrunningthesilentinstall
testsnamelyinstallsilentoracle<version>forsingleinstanceandinstallsilentracforaRAC
installation.(PleaserefertotheOracleLinuxTestdocumentfordetailsonthetestcases).

5.ExecutionofOLTTests
5.1.Schedulingtests

Toscheduletestsonthenodespecify:
oltschedulef <cmd>r
wherecmd=validate.cmdforOStesting

cmd=vcvalidate.cmdforvalidatedconfigurationtesting
11

Note:
Forsinglenodetests,ensurethattheinstallsilentoracle<>testcompletessuccessfullybeforeproceeding
toanyothertest.
ForRAC,ensurethattheinstallsilentractestcompletessuccessfullybeforeproceedingtoanyothertest.
Executethehugetlb/hugepagetestsseparatelywiththehugepagesetting,nr_hugepagesto800onyour
systemeitherthrough/proc/sys/vm/nr_hugepagesorthroughgrub.conf.Therestofthetestscanberun
withoutthissetting.

5.2.MonitoringTests
Tomonitortheprogressoftestsexecutedspecify(Thiscommandcanbeusedonlywhenthetestsare
executedandnotpriororpostexecution.)
oltschedulef validate.cmds
Awebbasedstatuspageisavailableathttp://<testnodename>/cgibin/status.pl
(Note:Apacheshouldbeinstalledandrunningforthisfeaturetowork)

5.3.Logsand TestResults
The/opt/oracle/oltest/oltschedule/log/directorywillhavenodespecificschedulinglogs.
The/opt/oracle/oltest/oltschedule/log/oltsummary.csvhasthecollatedresultsofalltheruns.
The/home/oracle/workhasthetestrunlogs.Arun.sucfileintheworkareaforthetestindicatessuccess.A
run.difintheworkareaforthetestindicatesfailure
ForRACrelatedtests,besidesthepresenceofarun.sucfileintheworkarea,pleasecheckforerrorsinthe
major_errorsfileinthisarea,forthesuccessofatest.

5.4.Terminatingtests
Forsinglenodetests,tocleanupalltestscripts,oracleprocess,otherderivativeprocessesandtestrelated
file/networklocks.
oltschedulek<testmachine_hostname>
ForRAC,tocleanupalltestscripts,oracleprocess,otherderivativeprocessesandtestrelatedfile/network
locksacrosseachoftheRACnodes,executethefollowingcommandfromthemasternode.
Oltschedulek<masternodename>z
NOTE:
.Ifamachinerebootsduringatest,doacleanuponallthenodesbeforerestartingatest.

12

5.5Resultsreporting

Collectandsubmitthefollowinglogsforauditing:
theworkareaforthetestin/home/oracle/work
/var/log/messages

/opt/oracle/oltest/oltschedule/log
ForRAC,inadditiontotheabovelogs,thefollowingarealsorequired
/opt/oracle/clusterware<version><architecture>/crs/log/<nodename>/
/tmp/crsctl*

Also/opt/oracle/oltest/oltschedule/utils/oltarchivelogsscriptcanbeusedtoarchivethe above
mentionedlogfilesautomatically.
i.e.cd/opt/oracle/oltest//oltschedule/utils
Tocollectsingleinstancetestlogs:

./oltarchivelogssi
TocollectRACtestlogs:
./oltarchivelogsrac
Executetheabovescriptonallthenodes&uploadthecollectedfile /home/oracle/work/olttestlogs
<hostname><timestamp>.tgz.fromallthenodesforresultauditing.

6.ExecutingRACdestructivetests
6.1.Runningautomateddestructivetests
Ifyouhaveconfiguredfortwonoderacwiththenodesnode1andnode2,thecmdfilefordestructivetests
willbe /opt/oracle/oltest/oltschedule/rac/<node1>_<node2>_dest_schedule.cmd
Thedestructivetestscanbescheduledinthefollowingmanner
$cd/opt/oracle/oltest/oltschedule
$oltschedulef rac/<node1>_<node2>_dest_schedule.cmdr

6.2.Runningmanualdestructivetests
Thisinvolvesrunninganormalractestusingtheoltframeworkandthendoingthedestructivepart
manuallyandthenobservingthebehavior.Thisisrequiredtobedonefornetworkfailureandstorage
failure.
13

1.Schedulethetests,usethefollowingcommand
$cd/opt/oracle/oltest/oltschedule
$oltschedulef rac/<node1>_<node2>_dest_manual_schedule.cmdr
2.Whentestracst<storage>1starts,waittillthetestshowsthefollowingmessagein

rac_tuning.tlgonthemasternode
$tailf /home/oracle/work/<currenttestrun>..../work/rac_tuning.tlg
"Formanualdestructivetestsmanuallydothefailurepart"
Thenperformoneofthefollowingtestscenarioonthenonmasternode,
a.Networkinterconnectfailure Failingtheprivateinterconnectonthenonmasternode
Youcanachievethisintwoways:
1.Bringdowntheprivateinterfaceusingtheifdowncommand
e.g.
$ifdowneth1
2.Pulltheinterconnectphysicallyorifyouhaveawebinterfaceforthenetworkswitchesthen
disablethecorrespondingswitchfromthewebinterface
(NOTE:Clusterreconfigurationdependsontheloadandenvironment)
Restoretheinducedfailurei.e.networkfailureoncethenodegoesdown.
3.Whentestracst<storage>2starts,waittillthetestshowsthefollowingmessageinrac_tuning.tlgon
themasternode
$tailf/home/oracle/work/<currenttestrun>..../work/rac_tuning.tlg
"Formanualdestructivetestsmanuallydothefailurepart"
Performthefollowingtestscenarioonthenonmasternode,
b.Storagefailure
Disablingthestorageonthenonmasternode
1.Pullthestoragecablephysicallyorifyouhaveawebinterfaceforthestorageswitchesthendisable
thecorrespondingswitchfromthewebinterface
Inthiscase,thenodeforwhichthestorageisdisabledisexpectedtobeevictedfromthecluster.
Generallythefollowingbehaviorisobservedinanyoftheabovedestructivetests:
Oneofthenodeswillgetevictedandtheothernodereconfiguresasacluster
Checkthecrslogsin/opt/oracle/clusterware<oracleversion><architecture>/crs/log/<node1>/cssd/ocssd.log
forclusterreconfigurationonbothnodes
14

Theconnectionsontheevictednodewillfailovertothesurvivingnodes.Youcancheckthisbychecking
therunprocessfilein/home/oracle/<workarea>
NOTE:
1)Therewillbefailuresreportedintheoracle/crslogsonthenodeonwhichthefailureisinduced.
Restoretheinducedfailurei.e.storagefailureoncethenodegoesdown.
2)Failurestobedoneasmentionedintheabovesection.Anyothermethodoffailureisnotrecommended.
3)Fororacleversion>11.2.0.2,duringmanualdestructivetests,theinducedstorageornetworkfailure
needstobereconnectedwithin510minsoffailure

6.3Runningmanualdestructivetests with failback

1.Schedulethetests,usethefollowingcommand
$cd/opt/oracle/oltest/oltschedule
edittherac/<node1>_<node2>_dest_manual_schedule.cmd&commentouttherestofthetestcases
retainingonlyracdest<storage>HA1testcase.
$oltschedulef rac/<node1>_<node2>_dest_manual_schedule.cmdr
2.Waittillthetest(racdest<storage>HA1)showsthefollowingmessageinrac_tuning.tlgonthemaster
node
$cd/home/oracle/work/<currenttestrun>./work/
$tailfrac_tuning.tlg
"Formanualdestructivetestsmanuallydothefailurepart"
Forautomateddestructivetestnowfailurewillhappen
Waitingforfailure[]min...
[NOTE:defaultis20Min.ThiscanbechangedbysettingMANUAL_FAILTIMEOUTinthe
testcaseenvfile]
3.Performthestoragefailurecondition(SimilartoStep3intheSectionRunningmanualdestructive
tests)

4.VerifybehavioroftheclusterasmentionedinStep4oftheSectionRunningmanualdestructivetests

15

5.Fororacleversion>11.2.0.2,duringmanualdestructivetests,theinducedstorageornetworkfailure
needstobereconnectedwithin510minsoffailure.Thenodemaynotrebootincaseofdatabase
version>11.2.0.2.
6.Checkrac_tuning.tlgfile.Itshouldshowthefollowingw.r.tthenode(usedforfailure/restoration)
1.

CRSstartup

2.

Databaseinstancestartup

3.

Neworacleconnectionsonthenode.Thiscanbealsoseenintherunprocesslog.

7.Verify/opt/oracle/clusterware<oracleversion><architecture>/crs/log/<node1>/cssd/ocssd.logonall
thenodesforreconfigurationoffailednode.
6.3.Resultsreporting
Collectandsubmitthelogsforauditingonallthe nodesusingthe /opt/oracle/oltest/oltschedule/utils/olt
archivelogsscriptasmentionedintheabovesection5.5

7.CleaninguptheOracle softwareinstallations
Shutdowncrsonallallnode,ifrunning.
Tocleanuptheinstallationscompletelythefollowingdirectoriesshouldberemoved:
1.oracle/crshome
e.g.
For10.2.0.3singlenodeoracle,
#rmrf/opt/oracle/oltest/.srchome/ora10.2.0.3x86_64/
For10.2.0.3rac,
#rmrf/opt/oracle/oltest/.srchome/orarac10.2.0.3/
#rmrf/opt/oracle/clusterware10.2.0.3x86_64
2.Inventorylocation
Removetheinventorylocationspecifiedin/etc/oraInst.loc
e.g.
For10.2.0.3singlenode/rac,
#cat/etc/oraInst.locinventory_loc=/opt/oracle/oltest/.srchome/orarac
10.2.0..3/oraInventoryinst_group=oinstall
#rmrf/opt/oracle/oltest/.srchome/orarac10.2.0.3/oraInventory
16

3.Incaseof11.2.0.1,11.2.0.2and11.2.0.3
.Bringdownanyvipinterfacesifitsup.
.Removeohasd/crsd/evmd/cssdentriesfrom/etc/inittab

8.DebuggingOLTissues
FordetaileddocumentationpleaserefertoDEBUG.txt under/opt/oracle/oltest/oltschedule/doc

8.AppendixA
8.1.ConfiguringOLT
StandardHostsconfiguration
OLTcanbeconfiguredintwoways:i.e.ManuallyandSilently;torunsilentlyruncommand
oltconfigureselseformanuallyruncommandoltconfigure.

Manual Configuration:
ChooseSingleinstanceorRACtestsfromthemenudependingonthetestsyouplantorun.
NOTE:Whenpromptedforthehostname,enterthefullhostname,includingthedomainname.
Duringconfiguration,
1.Forsingleinstancesetup:
Theuserispromptedtoenterrepositorylocation(nodeanddirectory)andtheroot
passwordsfortheinvolvednodes.Foralocalsystemthisremainsthecurrentmachinenameandthe
defaultrepositorylocation(/opt/oracle/oltest/oltrepository).Italsopromptsforthestoragetype.
2.ForRACSetup:
Theuserwillbepromptedfornodenames(public,privateinterconnectandvip,scanipfor
11gr2),storagetypeandlocation,ocrandvotelocations(optional),asmredundancylevel(optional)and
ocrandvoteredundancylevels,dnfs_clientfor11gR2.
1.Storagelocation
1.Forasmlib,providethelabelnames(ORCL:<label_name>)
2.Forasm,providethedevicenames(raw)forstoragelocation
3.ForNFS/OCFS2,providetheappropriatenameforthestoragelocation

17

2. OCR/Vote
1. For10gand11gR1,forasmandasmlibprovidetheocrandvotelocationsas
required.
2. For11gR2providenecessaryocrandvoteredundancylevelsandlocationtype(NAS
orASM)andtheircorrespondingocrandvotelocations.IncaseofASMlocation
provideocr/votelabelnameasORCL:XXX.
Note:Blockandrawdevicesforocr/votearenotsupportedin11gR2.

Theenvironmentvariablesattheendofsetuparesavedinenv/<HOSTNAME>/oltschedule.env.
Thisfilecanbeeditedtoaffectchangesinenvironment.Thestoragelocationscanbeeditedhere.
Ifusingtheadvancedsetup,OLTcanbeconfiguredusingthefollowingcommandoltconfigureG.

SilentConfiguration:
Tosilentlyconfiguremodifyalltherequiredparametersettingsin/opt/oracle/oltest/oltschedule/env/
oltconfigure.env(detailsontheparameters&settingsareavailableinthisfile).

9.AppendixB
Patchmatrixforanoracleproductversionforagivenplatform

OracleProduct
Database

Version

Architecture

10.2.0.2

X86_64

Patch Numbers
5071492,4639236,46907
94,5036588,5082668

Database

10.2.0.3

X86_64

5452672,6614205
5526245,5940792

Database

10.2.0.4

X86_64

7573282,7592346,76126
39,6079224,7715304,71

Database

11.1.0.6

X86_64

31656,5408688
7592335,6778860

Database

11.1.0.7

X86_64

7520217,7135702

10.AppendixCOLTUtilities
ThiscontainsthelistofOLTUtilitiesavailablein/opt/oracle/oltest/oltschedule/utils
1.

asmdebug/run[run(script)asoracleuserwithallasmdevicesascommandlinearguments]
Generatesasmtraces<node_name>.tgzfileswithdebugandtraceinformationsfordebugging
ASMrelatedissuesonRAC.
18

2. oltdebug[Noarguments]
ThisscriptcollectssystemconfigurationandlogfilesfordebuggingissuesrelatedtoOLTtest
executions.Generatesatarfileoltdebuglogs${time_stamp}.tgzinutilsdirectory
3.

oltisocopy[NoargumentsInteractive]
Thisscriptcopiesisoimagesdownloadedbytheuser(fromOracle)intoadirectorywhereOLT
expectstofindtheisoimage

4.

oltpatchcopy[NoargumentsInteractive]
Thisscriptcopiesoraclepatchfilesdownloadedbytheuser(frommetalink.oracle.com)intoa
directorywhereOLTexpectstofindthepatches

5.

oltlogcleaner[Noarguments]
Cleanuptestlogsforfreeingupspace.AllQAdirectoriesolderthan10dayswillbedeleted.No
archivewillbemaintained

6.

oltarchivelogs[si/rac/ovm]
Thisutilitycollectsolttestlogfilesandsystemlogsasanarchivefile
/home/oracle/work/olttestlogs`hostname`${time_stamp}.tgz.

7. startqa/stopqa[Noarguments]
Ascripttomanuallystartup/stopqamonitorwithoutrunningOLT.Usefulforcollectingsystem
logsduringdebugging

19