OracleRACVirtualization

InstallationGuide

AndreiDanielDumitru,AntonTopurov
andrei048@gmail.com,topurov@gmx.de
Supervisors:AntonTopurov,EricGrancher
OpenlabSummerStudentProgram2008
CERN,ITDES

9thSeptember2008


1.CreatingOracleVMmanagernode.........................................................................................................3
1.1OracleenterpriseLinuxinstallation....................................................................................................3
1.2OracleVMManagerinstallation.........................................................................................................3
1.3OracleVMManagerMigration(onlyifneeded).................................................................................6
2.CreatingOracleVMServernode.............................................................................................................6
2.1BIOSupdateprocedure:......................................................................................................................6
2.2InstallingtheOVMServer...................................................................................................................6
3.FindingtheVMServerandcreatingVMsonit.......................................................................................7
3.1CreatingaServerPool.........................................................................................................................7
3.2AddingaServer...................................................................................................................................8
3.3ImportinganISOFile.........................................................................................................................10
3.4CreatingaVirtualMachinefromInstallationMedia........................................................................11
4.ConfiguringtheOSforOracleuse.........................................................................................................14
4.1Installingadditionalpackages,neededbyOracle.............................................................................14
4.2SettinguptheNFSstorage................................................................................................................18
4.3UsingOracletemplates:....................................................................................................................19
5.InstallingCRS..........................................................................................................................................21
5.1SettingupSSHuserequivalence.......................................................................................................21
5.2CLUVFYutility....................................................................................................................................22
5.3InstallingCRS10.2.0.1:......................................................................................................................25
5.4Install10.2.0.3Patchset....................................................................................................................31
5.5Installadditionalpatches(optional).................................................................................................31
6.InstallingRDBMS....................................................................................................................................34
6.1InstallingOracleDatabase10g..........................................................................................................34
6.2Creatingthedatabase.......................................................................................................................36
7.Swingbechsettingsandtests.................................................................................................................37
8.PureXenSetup.......................................................................................................................................38
9.LiveMigration.........................................................................................................................................40
A.Virtualmachineinstallation(steps)order:.............................................................................................45
B.Thingstoknow.......................................................................................................................................46
Appendix.....................................................................................................................................................47
2

1.CreatingOracleVMmanagernode
1.1OracleenterpriseLinuxinstallation.

Somepicturesoftheinstallation:
http://www.iteye.nl/weblog/2006/10/27/screenshotsoracleenterpriselinuxinstall/
Guides:
http://www.oracle.com/technology/pub/articles/smiley11gr1install.html
http://www.oraclebase.com/articles/linux/OracleEnterpriseLinux5Installation.php
Appendix:

02OracleEntepriseLinuxInstallationScreenshots.pdf
03OracleEntepriseLinuxInstallationGuide.pdf
04InstallingOracleDatabase11gRelease1onEnterpriseLinux5(32and64bit).pdf

TheinstallationorOracleEnterpriseLinuxwasstraightforward,followingtheguideprovidedby
Oracle(thefirstlinkorappendix3).

Rootpassword:netapp2008
Myuser:andrei
Mypassword:netapp2008

1.2OracleVMManagerinstallation.

OracleVMManagerinstallsanOracle10gExpressEditionDatabaseusedforkeepingallits
recordsabouttheVMServer(s)andVM(s):

Specify the HTTP port that will be used for Oracle Application Express [8080]
Specify a port that will be used for the database listener [1521]

Installation:
sh runInstaller.sh
Welcome to Oracle VM Manager
Please enter the choice: [1|2|3]
1. Install Oracle VM Manager
2. Uninstall Oracle VM Manager
3. Upgrade Oracle VM Manager
1
Starting Oracle VM Manager installation ...
Prepare to install the Oracle XE database ...
Checking the supported platforms ... Done
Checking the prerequisite packages are installed ... Done
Checking the available disk space ... Done
Installing the oracle-xe-univ package (rpm) now ...

Done
Oracle Database 10g Express Edition Configuration
------------------------------------------------This will configure on-boot properties of Oracle Database 10g Express
Edition. The following questions will determine whether the database should
be starting upon system boot, the ports it will use, and the passwords that
will be used for database accounts. Press <Enter> to accept the defaults.
Ctrl-C will abort.
Specify the HTTP port that will be used for Oracle Application Express
[8080]:
Specify a port that will be used for the database listener [1521]:1531
Specify a password to be used for database accounts. Note that the same
password will be used for SYS and SYSTEM. Oracle recommends the use of
different passwords for each database account. This can be done after
initial configuration:
Confirm the password:
Do you want Oracle Database 10g Express Edition to be started on boot (y/n)
[y]:y
Starting Oracle Net Listener...Done
Configuring Database...Done
Starting Oracle Database 10g Express Edition Instance...Done
Installation Completed Successfully.
To access the Database Home Page go to "http://127.0.0.1:8080/apex"
Set default database schema to 'OVS'.
Please enter the password for account 'OVS':
Confirm the password:
Creating the Oracle VM Manager database schema ...Done
Installing the ovs-manager package (rpm) ...
Done
Installing the oc4j package (rpm) ...
Done
Please enter the password for account 'oc4jadmin':
Confirm the password:
Starting OC4J ... Done.
To access the OC4J Home Page and change the password go to
http://127.0.0.1:8888/em

Deploying Oracle VM Manager application to OC4J container.

Creating connection pool ... Done
Creating data source ... Done
Deploying application ... Done
Deploying application help ... Done
Configuring Oracle VM Manager DataCollector ... Done
Please enter the password for the default account 'admin':
Confirm the password:
Configuring SMTP server ...
Please enter the outgoing mail server (SMTP) hostname: smtp.cern.ch
Mail server checking, may need some time, please wait ...
Mail server 'smtp.cern.ch' check failed, want to re-enter it(y|N)?N
Setting the SMTP hostname server to smtp.cern.ch ...
Done
Please enter an e-mail address for account 'admin': andrei.dumitru@cern.ch
Confirm the e-mail address:andrei.dumitru@cern.ch
Failed to send mail to 'andrei.dumitru@cern.ch', want to re-enter the email
address(y|N)?N
Updating e-mail address for account 'admin' to 'andrei.dumitru@cern.ch' ...
Done
The console feature is not enabled by default.
For detailed setup, refer to Oracle VM Manager User's Guide
Installation of Oracle VM Manager completed successfully.
To access the Oracle VM Manager home page go to:
http://127.0.0.1:8888/OVS
To access the Oracle VM Manager help page go to:
http://127.0.0.1:8888/help/help

ThepasswordfortheaccountsSYSandSYSTEMforOracleDatabase10gExpressEditionis
oracle
Aftertheinstallationiscompleted,youcanenterintheOracleVMManagerwebinterface.

1.3OracleVMManagerMigration(onlyifneeded)

InordertocompareOracleVirtualizationwithlastyearsresults,OracleVMManagerwas
movedtoadifferentmachine,thuswehadtwoidenticalmachinesfortesting.Onewasrunning
OracleVMServerandtheotheroneOracleEnterpriseLinuxthatcomesalreadypatchedwith
Xen.
ThemigrationofOracleVMManagertoanothermachinewasdonefollowingthisinstructions:
http://download.oracle.com/docs/cd/E11081_01/doc/doc.21/e10901/appendix_b.htm#BGBJG
HDG
Appendix:15BackingUpandRestoringOracleVMManager.pdf

2.CreatingOracleVMServernode
2.1BIOSupdateprocedure:

1. DownloadnewBIOS:
http://h20000.www2.hp.com/bizsupport/TechSupport/SoftwareDescription.jsp?
lang=en&cc=us&prodTypeId=12454&prodSeriesId=3252512&prodNameId=3253418&swEnvOI
D=1093&swLang=13&mode=2&taskId=135&swItem=ir602351

2. Puttheimagefile7E3_0234.BINtoaUSBdisk
3. InserttheUSBinthecomputerandrebootthecomputer
4. PressF10toenterthesetupmenu
5. FromtheFilemenuchooseFlashSystemROM
6. SelecttheUSBandpressF10
7. SelectthefilethatyouwanttouseandpressF10

(Optional)Praythepowerwon'tshutoff.
8. BIOSupdatewillbeperformed
9. Rebootsothechangeswilltakeeffect.

AftertheBIOSupdate,toenablethehardwarevirtualizationgototheSecuritymenu,choose
OSSecurityandEnabletheVirtualizationTechnology.NowyouwillhaveVirtualization
hardwaresupport.
ThecomputersusedaretwoHPCompaqdc5750.Checkthespecificationfilesformoredetails.

2.2InstallingtheOVMServer
6


OracleVMAgent(ovsagent)Password:oracle
Rootpassword:netapp2008

Guide:http://www.oracle.com/technology/documentation/vm.html
Applendix:0108OracleVMServerInstallationGuideRelease2.1E11081_01.zip

Usingtheguidetheinstallationwentsmoothly,butwhenIwantedtologin,Oraclegave
awarningmessage:

"WARNING:YOURCPU(S)SUPPORTHARDWAREVIRTUALIZATIONBUTYOURSYSTEM
DOESNOT.
ENABLEVIRTUALISATIONINYOURBIOSORCONTACTYOURSYSTEMVENDOR."

WiththehelpofHPSupportIdidaBIOSupdatefromtheversion786E3v02.10to
version786E3v02.34.

3.FindingtheVMServerandcreatingVMsonit
3.1CreatingaServerPool
Tocreateaserverpool,performthefollowing:
1. IntheServerPoolspage,clickCreatePool.
2. EntertheOracleVMServerinformationand
addtheOracleVMServertotheserverpool.
YoucanaddmultipleOracleVMServerstoa
singleserverpool.
AserverpoolconsistsofatleastoneServer
PoolMaster,oneUtilityServer,andoneVirtual
MachineServerrespectively.Allcanbedonein
theallinoneconfiguration(seethefigure).

Enterthefollowingparameters:
o ServerHost/IP
Enterthehostname,orIPaddressof
theOracleVMServerthatfunctionsas
theServerPoolMaster.Forexample,
enterthefollowing:
192.168.1.1
or
hostname1.example.com
o ServerName
7

NametheServerPoolMaster.Thismustbeunique.
o AgentPassword
EnterthepasswordtoaccessOracleVMAgentinstalledontheServerPool
Master.

o ServerType
SelecttheOracleVMAgentoptions:
ServerPoolMaster
UtilityServer
VirtualMachineServer
IfyouselectUtilityServeryoumustentertheOracleVMAgentlogincredentials.
Theusermusthaveread/writeprivilegesforthe/OVSfolder.

o ServerLocation
ThelocationoftheOracleVMServer.Forexample,ServerRoom1.
Afteryouentertheparameters,clickAdd.Iftheinformationisincorrect,ortheOracle
VMServerisdown,youcannotaddit.
TheOracleVMServerisdisplayedintheServerstobeAddedtoServerPooltable.You
canselectit,andthenedititordeleteit.

3.2AddingaServer

ToaddaUtilityServer,oraVirtualMachineServertoanexistingserverpool,perform
thefollowing:
1.
2.
3.
4.

FromtheServerspage,clickAddServer.
Searchandselectaserverpool,andthenclickNext.
EntertheOracleVMServerparameters.
Confirmtheinformationyouhaveentered.
Thenextpicturesshowtheinformationofacreatedserver.


3.3ImportinganISOFile
DependingonwhereyourISOfilesarelocated,ontheVirtualMachineServer,oronother
computers,youcanimportISOfilesfromdifferentresourcesasfollows:
SelectingfromtheServerPool
Beforeimporting,copyISOfilestothefolder/OVS/iso_poolontheVirtualMachineServer.
ToselectanexistingISOfileontheServerPool,andregisteritinOracleVMManager,perform
thefollowing:
1. IntheResourcespage,clickISOFiles,andthenclickImport.
2. SelectSelectfromServerPool(Discoverandregister).
3. SelectthefollowingISOfileinformation:
o ServerPoolName
SelecttheserverpoolforwhichtheISOfileistobeused.
o ISOGroup
SelecttheISOgroup.OneISOgroupmaycontainmultipleISOfiles.
o ISOLabel
SelecttheISOfileyouwanttoimport.
4. Confirmtheinformation.
5. Afterimporting,thestatusoftheISOfileisPending.Youneedtoapproveittomakeit
availabletouse.
DownloadingfromExternalSource
TodownloadanISOfilefromoutsideoftheServerPool,suchasyourownHTTP/FTPserver,
performthefollowing:
1. FromtheResourcespage,clickISOFiles,andthenclickImport.
2. SelectDownloadfromExternalSource(HTTPandFTP).
3. EnterthefollowingISOfileinformation:
o ServerPoolName
SelecttheserverpoolforwhichtheISOfileistobeused.
o ISOGroup
EnterthenameoftheISOgroup.OneISOgroupmaycontainmultipleISOfiles.
o ISOLabel
EntertheISOlabelusedtoidentifytheimportedISOfile.
Forexample,enterCD1.
Note:
ThevalidISOlabelmustcomplywiththefollowingrules:
10

Theinitialletterofthelabelmustbeinuppercase,orlowercase.
Validcharacterswhichyoucanuseinclude,numbers(suchas1,2,3,and
soon),letters(fromatozandfromAtoZ),dash(),dot(.)and
underscores(_).
URL
EnterthepathwheretheISOfileislocated.YoucanimporttheISOfilethrough
anFTPserviceorthroughanHTTPwebservice.Forexample,enteranaddress
similartothefollowing:
http://host/el5_img/disc1.iso
ftp://host/el5_img/disc1.iso
Whenusingaproxy,selectUserProxyandentertheproxyaddress.Forexample,
enteraproxyaddresssimilartothefollowing:
http://host:port
4. Confirmtheinformation.
5. Afterimporting,thestatusoftheISOfileisPending.Youneedtoapproveittomakeit
availableforcreatingvirtualmachines.
o

StatusofISOFiles
TheimportedISOfilecanbeoneofthefollowingstatus:
Importing:ThisstatusindicatesthattheISOfileisintheprocessofbeingimported.
Pending:ThisstatusindicatesthattheISOfilehasbeenimportedsuccessfully,andis
waitingfortheapprovaloftheManager.
ImportError:Thisstatusindicatesthatthereweresomeerrorsduringtheimporting
process.ReimporttheISOfile,ordeleteit.
Active:ThisstatusindicatesthattheISOfilehasbeenapproved,andisavailablefor
creatingvirtualmachines.
Inactive:ThisstatusindicatesthattheISOfileisnotavailable.

3.4CreatingaVirtualMachinefromInstallationMedia
Apartfromanexistingtemplate,youcancreateavirtualmachinefrominstallationmedia,and
configuretheparametersmanually.
Tocreateavirtualmachinefrominstallationmedia,performthefollowing:
1. SelectCreatefrominstallationmedia.
2. Selecttheserverpoolwherethevirtualmachineistobelocated,andselectaVirtual
MachineServeronwhichthevirtualmachinewillrun.
SelectAutotoassignaVirtualMachineServerautomaticallyforthevirtualmachine,or
selectManualtospecifyoneormorepreferredserversmanuallyforthevirtual
machine.
3. Selectthevirtualizedmethod,FullyVirtualized,orParavirtualized.
o FullyVirtualized
SelectanISOfilefromthelist.
11

IfyourequiremorethanoneISOfiletocreatethevirtualmachine,selectthe
firstISOfile,andaftercreation,logontothevirtualmachine,andclickChange
CDtocontinuewiththeinstallation.
Inthefullyvirtualizedmethod,alsoknownasthehardwarevirtualizedmode
(HVM),theunmodifiedguestoperatingsystemrunsonthevirtualmachine.It
trapsandemulateseveryI/Oandhardwareinstruction.
Toapplythefullyvirtualizedmode,youmusthaveeitherIntelprocessorwithVT
(virtualizationtechnology)extensions,oranAMDprocessorwithSVM
extensions(alsocalledAMDV)availableonthehost.Acompletelistof
compatibleprocessorsisavailableat:
http://wiki.xensource.com/xenwiki/HVM_Compatible_Processors

IftheCPUdoesnotsupportHVM,usetheparavirtualizedmethodtocreatethe
virtualmachine.
o Paravirtualized
Beforeyoucreatethevirtualmachineusingtheparavirtualizedmethod,mount
allISOfilestoasinglemountpointontheVirtualMachineServerusingthe
followingcommand:
#mkdirmountpoint
#mountoloop,rocd1.isomountpoint
Where,mountpointreferstothedirectorytowhichyouwanttomountthe
files.
IntheResourceLocationfield,enterthefullpathofthemountpoint.HTTP,FTP,
andNFSaresupported.Forexample,enterapathsimilartothefollowing:
nfs:host:/mnt
Intheparavirtualizedmethod,theguestoperatingsystemisrecompiledbefore
beinginstalledonavirtualmachine.Also,thevirtualmachinedoesnotneedto
trapprivilegedinstructions.Trappingisamethodusedtohandleunexpected,or
unallowableconditions,whichistimeconsumingandcanimpactoperating
systemperformance.Withouttrappingprivilegedinstructions,the
paravirtualizedoperatingsystemrunsatnearnativespeed.
Currentlyfortheparavirtualizedmode,onlyOracleEnterpriseLinux4Update5,
andOracleEnterpriseLinux5aresupported.
4. Enterthefollowinginformationforthevirtualmachine:
o VirtualMachineName
Enterthenameofthevirtualmachine.Thismustbeunique.
o NumberofCores
Selectthenumberofcoresinthevirtualmachine.Youcanallocateamaximum
of32cores.
o MemorySize(MB)
12

Allocateatleast256MBofmemory.
o VirtualDiskSize(MB)
Allocateatleast1024MBofvirtualdisk.
o ConsolePassword
SettheconsolepasswordforconnectingtothevirtualmachineusingVNC.
o ConfirmConsolePassword
Reentertheconsolepassword.
5. Optionally,clickAddRowtoaddmorenetworkinterfacecards(NIC).Bydefault,
networkinterfacecardsarenamedVIF0,VIF1,VIF2,andsoon.Youcanrenamethem
aftercreatingthevirtualmachine.
TheMACaddressofanetworkinterfacecardisgeneratedautomaticallyandrandomly.
SelectaXenbridgeforthenetworkinterfacecard.Bydefault,thenumberofXen
bridgesisthesameasthatofphysicaladaptors,andXenbridgesarenamedafter
physicaladaptors.Forexample,iftheVirtualMachineServerhasseveralphysical
adaptors,eth0,eth1,andeth2,andsoon,Xenbridgesarethennamedxenbr0,xenbr1,
andxenbr2andsoon.
Whencreatingavirtualmachine,youcansetamaximumofthreenetworkinterface
cards.Afterthevirtualmachineiscreated,youcansetamaximumofeightnetwork
interfacecards.
6. Confirmthevirtualmachineinformationyouhaveentered.
7. Theprocessofcreatingavirtualmachinetakestime.ClicktheRefreshbutton
periodicallyuntilthestatusofthevirtualmachinechangesfromCreatingtoRunning.
Youcanchoosetorefreshmanually,ortorefreshevery30seconds.
Nowyoucanlogintothevirtualmachineandperformtheinstallation.

Afteravirtualmachinehasbeencreatedyoucanfollowallthestepsneededto
installCRSandanOracleDatabase(seetheOELinstallationchapter).Whenthemachine
isreadyforCRS,itcanbecloned,thusobtaininganothervirtualmachineidenticalwith
thefirstone.TheMACaddressesoftheNetworkInterface(s)willbeautomatically
changedduringtheclonningprocesssotherewillnotbeacollissionbetweentwo
identicalMACaddresses.Afterthat,youwillonlyneedtochangethehostnameofthe
newmachineandtheIPaddress(es)ifyouhaveassignedany.

TochangethehostnameofamachinerunningOracleEnterpriseLinuxjustedit
the/etc/sysconfig/networkfile
[oracle@oravm04 ~]$ cat /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=oravm04

TochangetheIP(s)ofanetworkinterfacecardeditthefileslocatedin
/etc/sysconfig/networkscripts/
13

[oracle@oravm03 ~]$ ls /etc/sysconfig/network-scripts/ | grep ifcfg-eth

ifcfg-eth0
ifcfg-eth1
ifcfg-eth2

4.ConfiguringtheOSforOracleuse
4.1Installingadditionalpackages,neededbyOracle

JusttomakesurewehaveallthepackagesneededfortheinstallationofCRSandthedatabase,
followthefollowinginstructions.Beforethat,youwillneedtoinstallfromthe1stcdthekernel
headers2.6.1892.el5.x86_64.rpmpackage.

ThepackagesareneededfortheinstallationoftheOracleDatabase11g:

64-bit (x86_64) Installations

Required kernel version: 2.6.18 This kernel, or any of the kernels supplied in updates, works
with Oracle Database 11g Release 1 .
Check your kernel version by running the following command:
uname -rm
Ex:
# uname -rm
2.6.18-8.el5 x86_64

Required package versions (or later):

binutils-2.17.50.0.6
compat-libstdc++-33-3.2.3
compat-libstdc++-33-3.2.3 (32 bit)
elfutils-libelf-0.125
elfutils-libelf-devel-0.125
gcc-4.1.1
gcc-c++-4.1.1
glibc-2.5-12
glibc-2.5-12 (32 bit)
glibc-common-2.5
glibc-devel-2.5
glibc-devel-2.5-12 (32 bit)
libaio-0.3.106
libaio-0.3.106 (32 bit)
libaio-devel-0.3.106
libgcc-4.1.1
libgcc-4.1.1 (32 bit)
libstdc++-4.1.1
libstdc++-4.1.1 (32 bit)
14

libstdc++-devel 4.1.1
make-3.81
sysstat-7.0.0
Note that there are a number of packages where both the 64-bit and 32-bit RPMs must be
installed. Fortunately, both are provided on the 64-bit Linux installation media.
If you've performed a "default RPMs" install as suggested by Oracle, there are still some
required packages that must be installed:
compat-libstdc++-33-3.2.3
compat-libstdc++-33-3.2.3 (32 bit)
elfutils-libelf-devel-0.125
gcc-4.1.1
gcc-c++-4.1.1
glibc-devel-2.5
glibc-devel-2.5-12 (32 bit)
libaio-devel-0.3.106
libstdc++-devel 4.1.1
sysstat-7.0.0
To install the remaining packages, mount the Oracle Enterprise Linux media and move to
the directory containing the RPMs.
Some required packages are dependent upon other packages, so the dependant packages
must be installed as well. Login as root and run the following commands to install the
remaining required packages and their dependent packages:

15

Mount CD 2
mount -t iso9660 -r /dev/cdrom /media
cd /media/Server
rpm -ivh elfutils-libelf-devel*.x86_64.rpm glibc-devel-2* gcc-4*.x86_64.rpm
gcc-c++-4*.x86_64.rpm libstdc++-devel-4*.x86_64.rpm glibc-headers*.x86_64.rpm
libgomp*.x86_64.rpm
Ex:
# rpm -ivh elfutils-libelf-devel*.x86_64.rpm glibc-devel-2* gcc-4*.x86_64.rpm
gcc-c++-4*.x86_64.rpm libstdc++-devel-4*.x86_64.rpm glibc-headers*.x86_64.rpm
libgomp*.x86_64.rpm
warning: elfutils-libelf-devel-0.125-3.el5.x86_64.rpm: Header V3 DSA
signature: NOKEY, key ID 1e5e0159
Preparing... ########################################### [100%]
1:glibc-headers ########################################### [ 11%]
2:glibc-devel ########################################### [ 22%]
3:libgomp ########################################### [ 33%]
4:gcc ########################################### [ 44%]
5:libstdc++-devel ########################################### [ 56%]
6:gcc-c++ ########################################### [ 67%]
7:glibc-devel ########################################### [ 78%]
8:elfutils-libelf-devel-s########################################### [ 89%]
9:elfutils-libelf-devel ########################################### [100%]
cd /
eject

Mount CD 3
mount -t iso9660 -r /dev/cdrom /media
cd /media/Server
rpm -ivh compat-libstdc++-33* libaio-devel*.x86_64.rpm
Ex:
# rpm -ivh compat-libstdc++-33* libaio-devel*.x86_64.rpm
warning: compat-libstdc++-33-3.2.3-61.i386.rpm: Header V3 DSA signature:
NOKEY, key ID 1e5e0159
Preparing... ########################################### [100%]
1:libaio-devel ########################################### [ 33%]
2:compat-libstdc++-33 ########################################### [ 67%]
3:compat-libstdc++-33 ########################################### [100%]
cd /
eject

Mount CD 4
mount -t iso9660 -r /dev/cdrom /media
cd /media/Server
rpm -ivh sysstat*
Ex:
# rpm -ivh sysstat*
warning: sysstat-7.0.0-3.el5.x86_64.rpm: Header V3 DSA signature: NOKEY, key
ID 1e5e0159
Preparing... ########################################### [100%]
1:sysstat ########################################### [100%]

To verify that the required packages have been installed on your system, run the following
command:
16

rpm -q --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n" binutils \

compat-libstdc++-33 elfutils-libelf elfutils-libelf-devel gcc gcc-c++ glibc \
glibc-common glibc-devel-2.5 libaio libaio-devel libgcc libstdc++ libstdc++devel \
make sysstat
Ex:
# rpm -q --queryformat "%{NAME}-%{VERSION}-%{RELEASE} (%{ARCH})\n" binutils \
> compat-libstdc++-33 elfutils-libelf elfutils-libelf-devel gcc gcc-c++ glibc
\
> glibc-common glibc-devel-2.5 libaio libaio-devel libgcc libstdc++
libstdc++-devel \
> make sysstat
binutils-2.17.50.0.6-2.el5 (x86_64)
compat-libstdc++-33-3.2.3-61 (i386)
compat-libstdc++-33-3.2.3-61 (x86_64)
elfutils-libelf-0.125-3.el5 (x86_64)
elfutils-libelf-devel-0.125-3.el5 (x86_64)
gcc-4.1.1-52.el5 (x86_64)
gcc-c++-4.1.1-52.el5 (x86_64)
glibc-2.5-12 (i686)
glibc-2.5-12 (x86_64)
glibc-common-2.5-12 (x86_64)
glibc-devel-2.5-12 (x86_64)
glibc-devel-2.5-12 (i386)
libaio-0.3.106-3.2 (x86_64)
libaio-0.3.106-3.2 (i386)
libaio-devel-0.3.106-3.2 (x86_64)
libgcc-4.1.1-52.el5 (x86_64)
libgcc-4.1.1-52.el5 (i386)
libstdc++-4.1.1-52.el5 (x86_64)
libstdc++-4.1.1-52.el5 (i386)
libstdc++-devel-4.1.1-52.el5 (x86_64)
make-3.81-1.1 (x86_64)
sysstat-7.0.0-3.el5 (x86_64)

ToinstallCRSandanOracleDatabasesomeextraconfigurationstepsarenecessary.

Createuseroracleandthegroupsciandoinstall:

[root@oravm03 ~]# groupadd ci

[root@oravm03 ~]# groupadd oinstall
[root@oravm03 ~]# useradd -m -g ci G oinstall oracle
[root@xenvmp04 ~]# id oracle
uid=500(oracle) gid=500(ci) groups=500(ci),501(oinstall)

17


4.2SettinguptheNFSstorage.

FortheDatabaseactualfiles,anNFSdirectorywassetuponanothermachine.
Guide:http://www.ubuntugeek.com/nfsserverandclientconfigurationinubuntu.html
Appendix:10NFSServerandClientConfigurationinUbuntu.pdf
NFSconfigurationfiles:

TheNFSserver:

andrei@pcora19:~$ cat /etc/exports

/OracleDatabase
137.138.34.1/8(rw,sync,no_root_squash,no_wdelay,anonuid=0,insecure_locks)

TheNFSclient:

137.138.34.28:/OracleDatabase /ORA/dbs03 nfs

hard,rw,noac,rsize=32768,wsize=32768,suid,proto=tcp,vers=3
137.138.34.28:/OracleDatabase /ORA/old nfs
hard,rw,noac,rsize=32768,wsize=32768,suid,proto=tcp,vers=3,actimeo=0,timeo=60
0 1 2
137.138.34.28:/OracleXEN /ORA/dbs03 nfs
rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,vers=3,timeo=600,noac
!!! /ORA/old is the location of the installation files

Becausetheinterfaceusedforinterconnectingthevirtualmachinesisapublicone,the
routingwasmanuallysetsothattheinterconnectinterfacesusedforcommunicationbetween
thevirtualmachinesandtheNFSserver.Thiswaythecommunicationisalwaysdonethrough
thedesiredinterfacejustlikeadedicatednetworkcardforthestorage.

ModifyedroutingtablesotheNFSstoragewillusetheappropriateinterfaces:
[root@oravm03 bin]# route -n
Kernel IP routing table
Destination
Gateway
137.138.34.28
*
137.138.0.0
*
137.138.0.0
*
169.254.0.0
*
10.0.0.0
*
default
default-route.c

Genmask
255.255.255.255
255.255.0.0
255.255.0.0
255.255.0.0
255.0.0.0
0.0.0.0

18

Flags
UH
U
U
U
U
UG

Metric
0
0
0
0
0
0

Ref
0
0
0
0
0
0

Use
0
0
0
0
0
0

Iface
eth2
eth0
eth1
eth2
eth2
eth1

Sothattheroutingisdoneeverytimethesystemboots,thefollowingcommandwasplacedin
rc.local:
routeaddhost137.138.34.28deveth2
[root@oravm03 bin]# cat /etc/rc.local
#!/bin/sh
#
# This script will be executed *after* all the other init scripts.
# You can put your own initialization stuff in here if you don't
# want to do the full Sys V style init stuff.
touch /var/lock/subsys/local
route add -host 137.138.34.28 dev eth2
[root@oravm03 bin]#

4.3UsingOracletemplates:

AlternativelytoinstallingandconfiguringVMs,onecanusepreformattedOracletemplates,availablefor
downloadfromedeliverywebsite.

Guide:
http://download.oracle.com/otn_software/virtualization/README.templates
Appendix:17README.templates.pdf

Ihavetriedtousethetemplates,butIcannotgettheOracleDatabase11gtomount
insidethevirtualmachineontheOracleEnterpriseLinux5system.

ItriedtousejusttheOracleEnterpriseLinuxtemplate,butthatdidnotwork,becauseI
havemodifiedtheRAMquantity(from2GBto1GB)andtheKernelgaveanerrormessage.
Maybebecauseithadafixedsizeswappartitioninsidedontknowforsure.

19


SoIhavedecidedtogotheclassicway,installOracleEnterpriseLinuxintheOracleVM
usingtheinstallationdisks.Theconsolefeaturehassomebugs(themouseintheconsoleisnot
themouseinthebrowser=youhave2cursors)

20

5.InstallingCRS
5.1SettingupSSHuserequivalence

BeforetheinstallationofCRScanbegin,someadditionalstepsarerequired.
First,youneedtoestablishuserequivalence.ThismeansthatoneachnodewhereCRSisgoingtobe
installedtheoracleusercanloginwithoutapassword,usingapublickey.

Wewillassumethatourclusterhasonlytwonodes:oravm03andoravm04.
ToconfigureSSHusingDSAonthatcluster,thefollowingstepsmustbetaken:
1.

Astheoracleuser,createthepublicandprivatekeysonbothnodes:

[oravm03]$/usr/bin/sshkeygentdsa

[oravm04]$/usr/bin/sshkeygentdsa

Acceptthedefaultlocationforthekeyfile.Whenpromptedforthepassphrase,just
presstheEnterkey.
2.Concatenatethecontentsoftheid_dsa.pubfilefromeachnodeintothe
authorized_keysfileonthefirstnode(/home/oracle/.ssh/authorized_keys):
[oravm03]$ ssh oravm03 "cat ~/.ssh/id_dsa.pub" >> \ ~/.ssh/authorized_keys
[oravm04]$ ssh oravm04 "cat ~/.ssh/id_dsa.pub" >> \ ~/.ssh/authorized_keys

3.Copytheauthorized_keysfiletotheothernodes:
oravm04$ scp ~/.ssh/authorized_keys oravm05:/home/oracle/.ssh/

21


/etc/hosts
EachnodemustknowallthehostsintheCRScluter;theknownmembersoftheCRScluster
mustbeaddedinthe/etc/hostsfileoneachnode.Youcanseeasampleofthe/etc/hostsfile
foraclusterwithtwonodes:oravm03andoravm04:

[oracle@oravm03 ~]$ cat /etc/hosts

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1
localhost.localdomain localhost
::1
localhost6.localdomain6 localhost6
134.138.34.16
ORAVM03.CERN.CH
ORAVM03
137.138.34.48
ORAVM03-STOR.CERN.CH
ORAVM03-STOR
137.138.34.49
ORAVM03-V.CERN.CH
ORAVM-V
10.1.1.3
ORAVM03-PRIV
###
137.138.34.50
ORAVM04.CERN.CH
ORAVM04
137.138.34.52
ORAVM04-STOR.CERN.CH
ORAVM04-STOR
137.138.34.51
ORAVM04-V.CERN.CH
ORAVM04-V
10.1.1.4
ORAVM04-PRIV

!!!Sonoconfusionwillexist,pleaseusethesamecase(eithersmallorBIGletters)forallthe
hostnames.

5.2CLUVFYutility

TocheckthatthesystemisreadyforinstallingCRScluvfycanbeused:
[oracle@oravm03 bin]$ ./cluvfy stage -post hwos -n oravm03,oravm04 verbose
Performing post-checks for hardware and operating system setup
Checking node reachability...
Check: Node reachability from node "oravm03"
Destination Node
Reachable?
------------------------------------ -----------------------oravm03
yes
oravm04
yes
Result: Node reachability check passed from node "oravm03".

22

Checking user equivalence...

Check: User equivalence for user "oracle"
Node Name
Comment
------------------------------------ -----------------------oravm04
passed
oravm03
passed
Result: User equivalence check passed for user "oracle".
Checking node connectivity...

Interface information for node "oravm04"

Interface Name
IP Address
Subnet
Gateway Hardware Address
---------------- ------------ ---------------------eth0
137.138.34.50 137.138.0.0
00:16:3E:37:97:6C
eth1
10.1.1.4
10.0.0.0
00:16:3E:2F:B4:5D
eth2
137.138.34.52 137.138.0.0
00:16:3E:1D:BC:D8

Interface information for node "oravm03"

Interface Name
IP Address
Subnet
Gateway Hardware Address
---------------- ------------ ---------------------eth0
137.138.34.16 137.138.0.0
00:16:3E:44:13:BF
eth1
137.138.34.48 137.138.0.0
00:16:3E:46:05:EF
eth2
10.1.1.3
10.0.0.0
00:16:3E:12:49:69

Subnet Gateway
------------

------------

0.0.0.0
UNKNOWN

137.138.1.1
137.138.1.1

Subnet Gateway

Default

------------

0.0.0.0

137.138.1.1

0.0.0.0

137.138.1.1

UNKNOWN

137.138.1.1

Check: Node connectivity of subnet "137.138.0.0"

Source
Destination
------------------------------ ---------------------------------oravm04:eth0
oravm04:eth2
oravm04:eth0
oravm03:eth0
oravm04:eth0
oravm03:eth1
oravm04:eth2
oravm03:eth0
oravm04:eth2
oravm03:eth1
oravm03:eth0
oravm03:eth1

23

137.138.1.1

0.0.0.0

------------

Default

Connected?
----------yes
yes
yes
yes
yes
yes

Result: Node connectivity check passed for subnet "137.138.0.0" with node(s)
oravm04,oravm03.
Check: Node connectivity of subnet "10.0.0.0"
Source
Destination
------------------------------ ---------------------------------oravm04:eth1
oravm03:eth2
Result: Node connectivity check passed for subnet "10.0.0.0" with
oravm04,oravm03.

Connected?
----------yes
node(s)

Interfaces found on subnet "137.138.0.0" that are likely candidates for VIP:
oravm04 eth0:137.138.34.50
oravm03 eth0:137.138.34.16
WARNING:
Could not find a suitable set of interfaces for the private interconnect.
Result: Node connectivity check passed.

Checking shared storage accessibility...

NFS
-----------------------------------/ora/dbs03

Sharing Nodes (2 in count)

-----------------------oravm04 oravm03

Shared storage check was successful on nodes "oravm04,oravm03".

Post-check for hardware and operating system setup was successful.

TheSubnetGatewayforthelocalinterfacesisUNKNOWNbecausethelocalinterfacesdonothave
configuredagateway.Althoughthereisawarningaboutnotfindingasetofsuitableinterfacesforthe
privateinterconnectduringtheinstallationprocesstherewasnoproblemrelatedtothatandduringthe
CRSinstallationalltheinterfaceswereassignedtheproperrole(public,private,storage).

24


5.3InstallingCRS10.2.0.1:

ToinstallCRSjustruntheinstallationscript:
[oracle@oravm03 ~]$sh /ORA/dbs03/clusterware/10.2.0.3/installer/runInstaller

Youcanseebellowsomeinstallationscreenshots:

25

SpecifyingtheCRSHomeDetails

26


CRSwillnowcheckifalltherequirementsfortheinstallationaremet.Youwillreceivesome
warningortheinstallerwillnotexecutetheCheckingoperatingsystemrequirementsand
Checkingoperatingsystempackagerequirements.ThisisbecausewehaveusedCRSversion
10withOracleEnterpriseLinux5.OEL5issupportedstartingfromCRSversion11.

27


HereyouwillhavetoconfigurethenodesonwhichCRSwillbeinstalled,usingtherequired
hostnamesforthepublic,privateandvipinterfaces.

28

Hereyoucanspecifytheroleofeachnetworkinterface.Thepublicinterfaceshouldbemarked
asPublic,thePrivateinterfaceasPrivateandthededicatedstorageinterfaceasDoNotUse.
UsetheEditbuttontochangetheconfigurationofeachinterface.

ChecktheCRSversison:
AftertheinstallationyoucanchecktheCRSversionwiththefollowingcommand:
[oracle@oravm03 bin]$ crsctl query crs activeversion
CRS active version on the cluster is [10.2.0.1.0]

29


ChecktheCRSisworkingonallnodes

[oracle@oravm03 bin]$ ./crs_stat -t

Name
Type
Target
State
Host
-----------------------------------------------------------ora....03.lsnr application
ONLINE
ONLINE
oravm03
ora....m03.gsd application
ONLINE
ONLINE
oravm03
ora....m03.ons application
ONLINE
ONLINE
oravm03
ora....m03.vip application
ONLINE
ONLINE
oravm03
ora....04.lsnr application
ONLINE
ONLINE
oravm04
ora....m04.gsd application
ONLINE
ONLINE
oravm04
ora....m04.ons application
ONLINE
ONLINE
oravm04
ora....m04.vip application
ONLINE
ONLINE
oravm04
ora.racdb.db
application
ONLINE
ONLINE
oravm03
ora....b1.inst application
ONLINE
ONLINE
oravm03
ora....b2.inst application
ONLINE
ONLINE
oravm04

PossibeVIPCAproblems:
IncaseyouhaveVIPcaproblems,checkoutthis:
Links:

https://metalink.oracle.com/metalink/plsql/f?p=130:14:6948147433366817732::::
p14_database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_f
ont:NOT,414163.1,1,1,1,helvetica
https://metalink.oracle.com/metalink/plsql/f?p=130:14:6948147433366817732::::p14_
database_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:
NOT,276434.1,1,1,1,helvetica
http://cs.felk.cvut.cz/10gr2/relnotes.102/b15659/toc.htm#CJABAIIF

Appendix:

11ModifyingtheVIPorVIPHostnameofa10gOracleClusterwareNode.pdf
1210gR2RACInstallissuesonOracleEL5orRHEL5orSLES10(VIPCAFailures).pdf
13SRVCTLandVIPCAUtilitiesSettheLD_ASSUME_KERNELParameter.pdf

AcommonvipcaproblemmyappearbecauseOELisnotsupporteddefaultbyCRS10.Thefirstlinkgives
thesolutionorappendix12.

30


5.4Install10.2.0.3Patchset

AfterCRS10.2.0.1.0theupgradeto10.2.0.3.0wasapplied(locatedin
/clusterware/10.2.0.3/patchset)
[oracle@oravm03 patchset]$ sh runInstaller

[oracle@oravm03 bin]$ ./crsctl query crs activeversion

CRS active version on the cluster is [10.2.0.3.0]

5.5Installadditionalpatches(optional)
Thepatcheslocatedinthe../clusterware/10.2.0.3/patchesdirectoryarespecificpatchesmade
byCERN.
Moreinformationaboutthem(1stlink)andinstallationinstructions(2ndlink)canbefoundhere:

https://twiki.cern.ch/twiki/bin/view/DESgroup/OracleCERNPatchLevels
https://twiki.cern.ch/twiki/bin/view/DESgroup/CreatingAnRpmOfOracleCrsandRACRdb
ms

CERNLevel1patches
RunningthedefaultcommanditsaysthatitdoesnotknowtheJDKlocationandspecifyingthe
JDKlocationreturnsanothererror.Seebelowfordetails:

[oracle@oravm03 OPatch]$ ./opatch lsinventory -detail -oh $RDBMS_HOME

ls: /ORA/dbs01/oracle/product/rdbms/jre: No such file or directory
Java could not be located. OPatch cannot proceed!
OPatch returns with error code = 1
[oracle@oravm03 OPatch]$ ./opatch lsinventory -detail -oh $RDBMS_HOME -jdk
/ORA/dbs01/oracle/product/crs/jdk/
java.lang.NoClassDefFoundError: oracle/ops/mgmt/cluster/ClusterException
at oracle.opatch.OPatchSession.<clinit>(OPatchSession.java:168)
at oracle.opatch.OPatch.<clinit>(OPatch.java:124)
OPatch failed with error code 255
[oracle@oravm03 OPatch]$

Solution
Link:https://metalink2.oracle.com/metalink/plsql/f?p=130:14:1864059879221541198::::p14_d
atabase_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,427
728.1,1,1,1,helvetica
31

Appendix:18OPatchJavaError.pdf

CERNLevel2patch
[oracle@oravm03 mega-6741817]$ /ORA/dbs01/oracle/product/crs/OPatch/opatch
apply -local -oh $CRS_HOME
Invoking OPatch 10.2.0.3.0
Oracle interim Patch Installer version 10.2.0.3.0
Copyright (c) 2005, Oracle Corporation. All rights reserved..

Oracle Home
: /ORA/dbs01/oracle/product/crs
Central Inventory : /ORA/dbs01/oracle/product/rdbms/oraInventory
from
: /etc/oraInst.loc
OPatch version
: 10.2.0.3.0
OUI version
: 10.2.0.3.0
OUI location
: /ORA/dbs01/oracle/product/crs/oui
Log file location :
/ORA/dbs01/oracle/product/crs/cfgtoollogs/opatch/opatch2008-07-30_18-2850PM.log
ApplySession applying interim patch '6741817' to OH
'/ORA/dbs01/oracle/product/crs'
You selected -local option, hence OPatch will patch the local system only.
Please shutdown Oracle instances running out of this ORACLE_HOME on the local
system.
(Oracle Home = '/ORA/dbs01/oracle/product/crs')
Is the local system ready for patching?
Do you want to proceed? [y|n]
y
User Responded with: Y
Final
Verifying the update...
Inventory check OK: Patch ID 6741817 is registered in Oracle Home inventory
with proper meta-data.
Files check OK: Files from Patch ID 6741817 are present in Oracle Home.
The local system has been patched and can be restarted.
OPatch succeeded.

32

Afterpatching(CERNpatchlevels1and2)theCRSwillnotstart:
CRS-0184: Cannot communicate with the CRS daemon.
Also,therearentanylogsin/ORA/dbs01/oracle/product/crs/log/
IremovedtheCRSusing:
Link:https://metalink.oracle.com/metalink/plsql/f?p=130:14:7095958152507896948::::p14_d
atabase_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,23
9998.1,1,1,1,helvetica
Appendix:1410gRACHowtoCleanUpAfteraFailedCRSInstall.pdf

Afterthis,acleanCRSinstallationwasdoneandpatchedtoCRS10.2.0.3versionwithout
applyinganyCERNpatches.

MisspellingError
WheninstallingCRS,thisErrorMessageapperead:

Thelocation<loc>enteredforOracleClusterRegistry(OCR)isnotsharedacrossallthenodesin
thecluster.
SOLUTION:CHECK/etc/hoststoseeifitiscorrectonallnodes!!!!!!!!!!
[inmycase,Ihadawrongdot(.)somewhereinthefile]

33

6.InstallingRDBMS

6.1InstallingOracleDatabase10g

Directorystructure:

/ORA/dbs00
/ORA/dbs01/oracle/product/rdbms(instancefiles)
/ORA/dbs03(databasefilesmountedNFSdirectoryfromanothercomputer)
/ORA/old(installationbinariesmountedNFSdirectoryfromanothercomputer)
ORABASE:/ORA/dbs01/oracle(ORACLE_BASE)
ORAHOME:/ORA/dbs01/oracle/product/rdbms(ORACLE_HOME)

Environmentvariables

TomakesurethatOraclesoftwarehasallthenecessaryenvironmentvariablessetupwhenthe
systemstartsascriptisexecutedwhenthesystemshellisstartedbytheoracleuser:
[oracle@oravm03 ~]$ cat .bashrc
# .bashrc
# Source global definitions
# User specific aliases and functions
source set_racdb1_database;

Thecontentsofthefile:
[root@OraVM01 oracle]#source set_racdb1_database
[root@OraVM01 oracle]# cat set_oravm_database
ORACLE_BASE=/ORA/dbs01/oracle; export ORACLE_BASE
ORACLE_PRODUCT=$ORACLE_BASE/product; export ORACLE_PRODUCT
ORACLE_HOME=$ORACLE_PRODUCT/rdbms; export ORACLE_HOME
ORACLE_SID=racdb1; export ORACLE_SID
TNS_ADMIN=$ORACLE_HOME/network/admin export TNS_ADMIN
LD_LIBRARY_PATH=$ORACLE_HOME/lib; export LD_LIBRARY_PATH
PATH=$PATH:$ORACLE_HOME/bin:$ORACLE_HOME; export PATH

Youcanfindtheset_ovm_databasefileintheappendixdirectory.
34


Installation

AftertheinstallationofOracleEnterpriseLinux5Release2,ItriedtoinstallOracle10g,but
OracleDatabase10gdoesnotworkdirectlywithOracleEnterpriseLinux5,becauseOELwas
releasedaftertheDatabase10g.

Starting Oracle Universal Installer...

Checking installer requirements...
Checking operating system version: must be redhat-3, SuSE-9, redhat-4,
UnitedLinux-1.0, asianux-1, asianux-2 or redhat-5
Failed

SeemetalinkNote:456634.1(Applendix:06MetalinkNote456634.1.pdf)fordetails:
https://metalink.oracle.com/metalink/plsql/f?p=130:14:6948147433366817732::::p14_databas
e_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,456634.1,1
,1,1,helvetica

Afterthat,camethis:

35


Seemetalink.oracle.comNote:443617.1(07MetalinkNote443617.1.pdf)fordetails:
https://metalink.oracle.com/metalink/plsql/f?p=130:14:6948147433366817732::::p14_databas
e_id,p14_docid,p14_show_header,p14_show_help,p14_black_frame,p14_font:NOT,443617.1,1
,1,1,helvetica

Basically,therewasaneedforanotherpackage,libXp1.0.08.1.el5.i386.rpm.

Yes,itisapackagefora32bitmachine!!!
Afterthat,theinstallationwentnormallyandtheupdateaswell.
TheRDBMSwasupdatedfrom10.2.0.1.0to10.2.0.3.0usingthepatchlocatedin
(../database/10.2.0.3/patchset).
TheCERNpatches(../database/10.2.0.3/patches)werenotapplied.
Checkingdatabaseversion
$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.3.0 - Production on Wed Sep 24 23:05:57 2008
Copyright (c) 1982, 2006, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining
options

6.2Creatingthedatabase

Afterthesoftwarewasinstalledthedatabasewascreatedusingdbca.
[oracle@OraVM01 ~]$ source set_oravm_database [incaseyoudidnotconfiguredthe
environmentyet]
[oracle@OraVM03 ~]$ dbca

Important
Whencreatingthedatabaseusethefollowingvalues(tohaveanaccuratecomparisonwiththe
previousyearsresults):
SGA:512MB
PGA:272MB
RedoLogs:1GB

36

7.Swingbechsettingsandtests
Swingbenchwastheonlytoolusedfortestingtheperformanceofthedatabase.
TwotestswereusedforOrderEntry(OE)andStressTest(ST).Bothtestshavebeendonein
LoadBalancing(LB)andNoLoadBalancing(NLB)mode.
FortheLBmode,inthetnsnames.orafilewasspecifiedLOAD_BALANCE = yes.
FortheNLBmode,2Swingbenchinstanceswereused,eachinstancetestingadifferentnode.
YoucandfindintheAppendixdirectoryalltheconfigurationandtestfiles.

37

8.PureXenSetup
AftersuccessfulmovingtheVMManager,OELwasinstalledonthemachinethatwillhostOEL
XENvirtualization.Theinstallationwasnormal.
Forcreatingthevirtualmachinesvirtmanager(GUIprovidedbyRHEL)wasused.
Tofinishtheinstallation,theinstallerhadtoreboot...theXENwillnotstart.
Theproblemisthat,ifyoufollowedtheinstruction,youhaveremovedthecdrom(isoiamge
file)andthemachineissetuptobootfromcdrom.
Edittheconfigfile/etc/xen/xenvm01(/etc/xen/<your_vm_name>)andspecifyboot="c"(to
bootfromtheharddrive).

Checkthisformoredetailsonconfigurationoptions(section:Configurationfileforunmodified
HVMguests):
http://www.cl.cam.ac.uk/research/srg/netos/xen/readmes/user/user.html#SECTION04320000
000000000000

Appendix:16XenManual.pdf

After the machine has rebooted, the

installation worked fine until I tryed
tousexmconsolexenvm01.
This command hangs and does not
work. I never managed to make this
work.
After the machine obtained an IP
addressIcouldsshtoitanduservirt
manager.
Atfirst,virtmanagerwouldnotwork.
Aftercreatingandsettingupthevirtualmachine,Iwasabletousevirtmanager(pictureabove)
utility just enough to add 2 extra network cards to the virtual machine. After that, the virt
managerwillnotdoanythinguponconnectingtothelocalhost.
Thelog(/root/.virtmanager/virtmanager.log)output:

[Wed, 27 Aug 2008 16:59:12 virt-manager 32586] ERROR (virt-manager:132)

Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/engine.py", line 185, in
_do_show_console
self.show_console(uri, uuid)

38

File "/usr/share/virt-manager/virtManager/engine.py", line 239, in

show_console
con.get_vm(uuid))
File "/usr/share/virt-manager/virtManager/connection.py", line 278, in
get_vm
return self.vms[uuid]

KeyError: '2a77f13e-a5ee-d73f-b612-69e8ab35e73c'
None

So,Iusedvirtviewer<machineid/name>andIcouldseetheVNCinterface.
MaybeitwasbecauseoftheXserverimplementationIusedfromtheWindowsVistamachineI
wasworkingon.Thereisachancethattherewillnotbeanyproblemsifanauthenticlinuxis
usedwithXforwarding.
AfterperformingallthenecessarystepsinpreparingthesystemforOracleinstallation(adding
more network interfaces, creating the oracle user and the group ci, setting the kernel
parameter, installing the required packages) I cloned the machine using virtclone. If no
parametersaresupplied,itwillstartininteractivemode.

[root@pcitdes013 ~]# virt-clone

What would you like to use as the cloned disk (file path)?
/var/lib/xen/images/xenvm02.img
What is the name or uuid of the original virtual machine? xenvm01
What is the name for the cloned virtual machine? xenvm02
Cloning from /var/lib/xen/images/xenvm01.img to
/var/lib/xen/images/xenvm02.img

Thecloningprocesswassuccessful.
Thefirstcreatedmachinesweredoneusinghardwarevirtualizationandafterallthetestingwas
donewemovedtoparavirtualization.
BecauseweranoutofIPswestartedusinglocalIPs.
At some point, CRS complained about a gateway that did not exist because we were using
localipssoIusedmasqueradingandsetthemachineIwasworkingonasagateway.
Sharing Internet Connection in Ubuntu: http://www.ubuntugeek.com/sharinginternet
connectioninubuntu.html
Appendix:19SharingInternetConnection.pdf

39

9.LiveMigration

Setup

Inordertotestlivemigration,IusedOracleVMServerrunningonmultiplemachines(3servers
withOracleVMServer).ForLiveMigrationtowork,theserversneedtohaveacommonstorage.The
pssibilitiesareanNFSClusteroriSCSI.FormytestIhaveusedaniSCSIcluster.

ForiSCSItoworkyouneedtheiSCSIInitiatorUtils[fromWikipedia:Aninitiatorfunctionsasan
iSCSIclient.AninitiatortypicallyservesthesamepurposetoacomputerasaSCSIbusadapterwould,
except that instead of physically cabling SCSI devices (like hard drives and tape changers), an iSCSI
initiatorsendsSCSIcommandsoveranIPnetwork.]

The iSCSI initiator is provided by the iscsiinitiatorutils6.2.0.8680.7.el5.x86_64.rpm package

fromtheOracleEnterpriseLinuxcds.

Also, for the actual storage an iSCSI target [from Wikipedia: iSCSI refers to a storage resource
locatedonaniSCSIserver(moregenerally,oneofpotentiallymanyinstancesofiSCSIrunningonthat
server)asa"target".AniSCSItargetusuallyrepresentsharddiskstorage.Aswithinitiators,softwareto
provide an iSCSI target is available for most mainstream operating systems.] is needed. In my cause I
usedOpenfiler(http://www.openfiler.com/)tocreateaniSCSItarget.

After I created the target I used the instructions from Oracle VM Server User's manual to
connecttheserverstothestorage.

Since
this
is
a
shared
storage
,
a
clustered
file
system
[http://en.wikipedia.org/wiki/Clustered_file_system]wasneeded.

Oracle provides OCFS2 [From http://oss.oracle.com/projects/ocfs2/ : OCFS2 is a POSIX

compliantshareddiskclusterfilesystemforLinuxcapableofprovidingbothhighperformanceandhigh
availability.Asitprovideslocalfilesystemsemantics,itcanbeusedwithanyapplication.Clusteraware
applicationswillbeabletomakeuseofparallelI/Oforhigherperformance.Otherapplications,ifnot
abletobenefitfromparallelI/O,willbeabletomakeuseofthefilesystemtoprovideafailoversetup
toincreasetheavailabilityoftheapplication.]

Following instructions provided by Oracle VM Server's Guide i was able to make all the server have a
commonviewofthestorage:

40


CreatingaSharedVirtualDiskforLiveMigration

Ifyouwanttoperformlivemigrationofdomainstoother,identical,computers,youmustcreate
asharedvirtualdisktobeusedduringthelivemigration.Youcansetupasharedvirtualdiskinthe
followingconfigurations:

OCFS2(OracleClusterFileSystem)usingtheiSCSI(InternetSCSI)networkprotocol
*

OCFS2usingSAN(StorageAreaNetwork)
*

NFS(NetworkFileSystem)

Thissectiondiscussescreatingasharedvirtualdiskintheseconfigurations.
6.1.1CreatingaSharedVirtualDiskUsingOCFS2oniSCSI

TocreateasharedvirtualdiskusingOCFS2oniSCSI:

1.InstalltheiscsiinitiatorutilsRPM.TheiscsiinitiatorutilsRPMisavailableontheOracleVMServer
CDROMorISOfile.

rpmUvhiscsiinitiatorutils6.2.0.7420.5.el5.i386.rpm

2.StarttheiSCSIservice:

serviceiscsistart

3.RundiscoveryontheiSCSItarget.Inthisexample,thetargetis10.1.1.1:

iscsiadmmdiscoverytsendtargetsp10.1.1.1

Thiscommandreturnsoutputsimilarto:

10.1.1.1:3260,5iqn.199204.com.emc:cx.apm00070202838.a2
10.1.1.1:3260,6iqn.199204.com.emc:cx.apm00070202838.a3
10.2.1.250:3260,4iqn.199204.com.emc:cx.apm00070202838.b1
10.1.0.249:3260,1iqn.199204.com.emc:cx.apm00070202838.a0
41

10.1.1.249:3260,2iqn.199204.com.emc:cx.apm00070202838.a1
10.2.0.250:3260,3iqn.199204.com.emc:cx.apm00070202838.b0

4.Deleteentriesthatyoudonotwanttouse,forexample:

iscsiadmmnodep10.2.0.250:3260,3Tiqn.199204.com.emc:cx.apm00070202838.b0odelete
iscsiadmmnodep10.1.0.249:3260,1Tiqn.199204.com.emc:cx.apm00070202838.a0odelete
iscsiadmmnodep10.2.1.250:3260,4Tiqn.199204.com.emc:cx.apm00070202838.b1odelete
iscsiadmmnodep10.1.1.249:3260,2Tiqn.199204.com.emc:cx.apm00070202838.a1odelete
iscsiadmmnodep10.0.1.249:3260,5Tiqn.199204.com.emc:cx.apm00070202838.a2odelete

5.Verifythatonlythenecessaryentriesarevisible:

iscsiadmmnode

6.Reviewthepartitionsbychecking/proc/partitions:

cat/proc/partitions
majorminor#blocksname
8071687372sda
81104391sda1
8271577607sda2
253070516736dm0
25311048576dm1

7.RestarttheiSCSIservice:

serviceiscsirestart

8.Reviewthepartitionsbychecking/proc/partitions.Anewdeviceislisted.

cat/proc/partitions
majorminor#blocksname
8071687372sda
81104391sda1
8271577607sda2
253070516736dm0
25311048576dm1
8161048576sdb

9.Thenewdevicecannowbeused.

42

fdiskl/dev/sdb

10.Createanewdirectorynamed/etc/ocfs2directory:

mkdir/etc/ocfs2

11.CreatetheOCSF2configurationfileas/etc/ocfs2/cluster.conf.Thefollowingisasample
cluster.conffile:

node:
ip_port=7777
ip_address=10.1.1.1
number=0
name=example1.com
cluster=ocfs2
node:
ip_port=7777
ip_address=10.1.1.2
number=1
name=example2.com
cluster=ocfs2
cluster:
node_count=2
name=ocfs2

12.ReviewthestatusoftheOCFS2clusterservice:

serviceo2cbstatus

13.LoadtheOCFS2module:

serviceo2cbload

14.SettheOCFS2servicetobeonline:

serviceo2cbonline

15.ConfiguretheOCFS2servicetostartautomaticallywhenthecomputerboots:

serviceo2cbconfigure

16.StartuptheOCFS2service.
43


serviceo2cbstart

17.[ONLYDOTHISSTEPFORONENODE]FormatthesharedvirtualdiskfromANYoftheserversinthe
cluster:

mkfs.ocfs2/dev/sdb1

18.MountthesharedvirtualdiskfromALLtheserversintheclusteron/OVS/remote:

mount/dev/sdb1/OVS/remote/tocfs2

[IMOUNTED/OVSDIRECTLY,NOT/OVS/remote]

19.Changethe/etc/fstabfiletoincludethesharedvirtualdiskmountedatboot:

/dev/sdb1/OVS/remoteocfs2defaults10

AftertheOracleVMServershavebeensetupIcreatedmultiplevirtualmachinesandusedthe
LiveMigrationfeaturefromtheOracleVMManager.Thevirtualmachine(s)Ichoosetomigratewas
movedfromtheserveritwasrunningon,toanotherserver.DuringtheLiveMigrationIkeeptPINGing
thevirtualmachineanditrespondedwithoutloosinganypackages.

Nextstepwastotestavirtualmachinerunningasingledatabase.Iinstalledasingleinstance
OracleDatabaseusingtheinstructionsprovidedinthisinstallationguide.AfterthatIstartedrunningan
OrderEntrytestusingSwingbench.Duringthetest,themachineperformedaLiveMigration.Thisdid
notaffecttheOrderEntrytestandSwinbenchperformedasusual.

44

A.Virtualmachineinstallation(steps)order:
VMpart:
1. createvirtualmachine
2. createORACLEuser
3. createdirectorystructureforthedatabaseasoracleuser(/ORA/)
4. mountNFS
5. installrequiredpackagesandsetrequiredparameters(followtheguide)

Oraclepart:
1.InstallCRS
(+patchit)
2.InstallRDBMS
(+patchit)
3.Createthedatabase

WhencloningVMs:
changehostnameofthenewVM
edit/etc/hostswiththeIPsandhostnames
makeoracleuserequivalencyonallnodes(sshwithkeys)

45

B.Thingstoknow
Command

Description

StartOracleEnterprise
Manager

./emctlstartdbconsole
./emctlstopdbconsole
Sqlplus>select*fromGV$instanc
Sqlplus>select*fromV$instance

Startupthedatabase

Sqlplus>startup

Shutdownthedatabase

Sqlplus>shutdown

GetCRSstatus

./crs_statt

./srvctl config database -d racdb

./lsnrctlstatus
crsctlquerycrssoftwareversion
crsctlquerycrsactiveversion
select*fromv$versionwherebannerlikeOracle%
Sqlplus/assysdba
./servctlstop/startdatabase<name>
./runcluvfy.shcompssannode_list

46

Listenerstatus
GetCRSversion
DatabaseSoftwareversion
Anotherwaytoseethe
databasesoftwareversion
Start/Stopdatabase
Checkforstorage


Appendix
Filename
0108OracleVMServerInstallationGuideRelease2.1E11081_01.zip

Description
OracleVMInstallationGuide
(forOracleVMServerand
Manager)
02OracleEntepriseLinuxInstallationScreenshots.pdf
SomeOracleEnterpriseLinux
InstallationScreenshoots
03OracleEntepriseLinuxInstallationGuide.pdf
AnOracleEnterpriseLinux
InstallationGuide
04InstallingOracleDatabase11gRelease1onEnterpriseLinux5(32 OfficialInstallGuideForOracle
and64bit).pdf
Database11gR1andOracle
EnterpriseLinux5
sp39197.exe
Originalfiledownloadedfrom
theHPwebsite.Itcontainsthe
SP39197directory.
SP39197(directory)
Allthefilesnecessarytoupdate
theBIOS
SP39197/BiosFlash.htm
BIOSUpdateProcedures
05README.templates
Thisdocumentcontains
informationoninstallingthe
OracleVMtemplates.

06MetalinkNote456634.1.pdf
Solutiontotheoperating
systemcheckfail.
07MetalinkNote443617.1.pdf
Solutiontothejavaerror
receivedwhenrunningthe
installerthemissingpackage
islibXp
08InstallationofOracle10g.pdf
AninstallationofOracle
Database10gonCentOS
usefulforthecommon
mistakes(@endoftheguide)
09set_oravm_database
Setstheenvironmentvariables
fortheOracleDatabase
10NFSServerandClientConfigurationinUbuntu.pdf
ConfiguringtheNFSserveron
UbuntuLinux
11ModifyingtheVIPorVIPHostnameofa10gOracleClusterware
ModifyingtheVIPorVIP
Node.pdf
Hostnameofa10gOracle
ClusterwareNode
1210gR2RACInstallissuesonOracleEL5orRHEL5orSLES10(VIPCA Wheninstalling10gR2RACon
Failures).pdf
OracleEnterpriseLinux5or
RHEL5orSLES10thereare
47

13SRVCTLandVIPCAUtilitiesSettheLD_ASSUME_KERNEL
Parameter.pdf
1410gRACHowtoCleanUpAfteraFailedCRSInstall.pdf

15BackingUpandRestoringOracleVMManager.pdf

16XenManual.pdf
17README.templates.pdf
18OPatchJavaError.pdf
19SharingInternetConnection.pdf

48

threeissuesthatusersmustbe
awareof
SomemoreVIPCAandSRVCTL
debuginformation
Thepurposeofthisdocument
istohelpDBA'sandsupport
analystsunderstandhow
tocleanupafailedCRS(Cluster
ReadyServices)installfor10g
RAC.

Howtobackupandrestore
OracleVMManager(goodfor
movingittoanothermachine)
XenManual
OracleVMtemplates
information
OPatchJavaErrorSolutionfrom
Metalink
SharingInternetConnectionin
Ubuntu