OracleRAC12c

DatabaseonLinux
UsingVirtualBox

lab128
LikePage

419likes

Bethefirstofyourfriendstolikethis

BySergeiRomanenko
October,2013
ThisarticledescribestheinstallationofOracleDatabase12cRelease1(12.1.0.164bit)RAConLinux
(OracleLinux6.464bit)usingVirtualBox(4.2.18).Highlightsofthisinstallationprocedure:
SmallestpossiblefootprintintermsofRAManddiskspacewithoutnoticeableperformancehit
ShortandfastwaytogetRACinstalledandrunning.Onlyrequiredoperationsincluded.Ifyou
canfindmoreefficientway,pleaseletmeknow
BothASMLibandUdevshareddiskconfigurationsarecovered
GUImaximized,itisfriendlyforbeginners,savestimeforexperiencedveterans

Content
Introduction
SystemRequirements
DownloadSoftware
VirtualMachineSetup
GuestOperatingSystemInstallation
CheckInternetAccess
OracleClusterwareInstallationPrerequisites.Part1
InstallGuestAdditions
OracleClusterwareInstallationPrerequisites.Part2
NetworkSetup
DownloadedOracleInstallationFiles
ClonetheVirtualMachine
CreateSharedDisks
InstalltheGridInfrastructure
InstalltheDatabaseSoftware
InstalltheDatabase
ChecktheStatusoftheRAC
MakingImagesoftheRACDatabase
RestoringRACfromSavedFiles

Introduction

ThisismysecondarticleonOracleRACinstalledinVirtualBox.ThefirstonewasaboutRACversion
11g(../rac_installation_using_vb/article_text.html),whichdrewverypositivefeedbackfromreaders.
Overlastyear,IbecameveryactiveuserofVirtualBox,thisisquitestablepieceofsoftwareandI
movedmanyagingphysicalcomputersinmyhouseholdintovirtualmachines.AyearagoIbought16
GBofRAMformylaptop(payingabout$100)andneverregrettedaboutthispurchase.Probablyitis
evenmoreaffordablenow.GiventhistrendofgallopingRAMsize,theentrylevellaptopcomputers
with16GBarenotfarawayfromnow.InthisarticleImadeadjustmentsforOracle12candlatest
VirtualBoxversions.Somestepswerestraightenedordroppedmakinginstallationof12cslightlyeasier
than11g.Yourfeedbackiswelcome,ithelpstomakethewholeproceduresimplerandfaster.
IfyouwanttogetthroughallstepsoftheOracleRACinstallationandyourlaptopordesktopcomputer
has8GBormoreofRAM(16GBisrecommended),thenthisisentirelyfeasibleusingOracle
VirtualBoxasdemonstratedinthisarticle.YoucangetarunningRACsystemwhichcanhostasmall
testdatabase.Thecreatedsystemisnot,andshouldnotbeconsidered,aproductionreadysystem.
It'ssimplytoallowyoutogetusedtoinstallingandusingRACandtestvariousadministration
procedures.ThearticlealsoexplainshowtosavetheimagesandrestoreRACfromtheimagesina
matterofminutes.Evenifyoubreakyourtestsystem,itwillbeeasytorestore.
Thisarticleusesthe64bitversionsofOracleLinux,version6.4,andOracle12cRelease1,version
12.1.0.1.0.UsingVirtualBoxyoucanrunmultipleVirtualMachines(VMs)onasingleserver,allowing
youtorunbothRACnodesonasinglemachine.Inaddition,itallowsyoutosetupsharedvirtualdisks.
Thefinishedsystemincludestwoguestoperatingsystems,twosetsofOracleGridInfrastructure
(Clusterware+ASM)andtwoDatabaseinstancesallonasingleserver.

SystemRequirements
8GBofRAM
44GBoffreespaceontheharddisk+stagingareaforOracleLinuxandRACdownloads,about
9GB
Thisprocedurewastestedon64bitWindows7.Thesuccessfulinstallationwasalsoreportedon
MacBookPro2011,8GBRAM,withOSX10.8.5(MountainLion)andVirtualBox4.2.18thanks
toRudreshRanawhosharedthisinformation.ItshouldbenoproblemusingVirtualBoxonother
HostOSes.PleaseletmeknowifyouhadsuccessorproblemsinotherOSes

DownloadSoftware
Downloadthefollowingsoftware.
OracleLinux(http://edelivery.oracle.com/linux)OnceintheOracleCloudservice,select"Oracle
Linux"and"x8664bit".Thenclickonrelease(6.4.0.0.0)anddownloadOracleLinuxRelease6
Update4forx86_64(64Bit)V3708401
VirtualBox(http://www.virtualbox.org/wiki/Downloads)(thelatestavailableversion4.2.18was
used)
OracleDatabase12cRelease1(12.1.0.1.0)
(http://www.oracle.com/technetwork/database/enterpriseedition/downloads/index.html)(use
Linuxx8664,then"SeeAll"link).
OracleDatabase12cRelease1GridInfrastructure(12.1.0.1.0)forLinuxx8664
(http://www.oracle.com/technetwork/database/enterpriseedition/downloads/index.html)(use
Linuxx8664,then"SeeAll"link).

VirtualMachineSetup
Inthisexercise,weareusingVirtualBoxinstalledon64bitWindows7.
WemustdefinetwovirtualRACnodes.WecansavetimebydefiningoneVM,thencloningitwhenitis
installed.
StartVirtualBoxandclickthe"New"buttononthetoolbar.Enterthename"rac1",OS"Linux"and
Version"Oracle(64bit)",andthenclickthe"Next"button:

SetBaseMemoryto3072MB,thenclickthe"Next"button:

Acceptthedefaultoptiontocreateanewvirtualharddiskbyclickingthe"Create"button:

AcceptthedefaultVDItypeandclickthe"Next"buttonontheCreateVirtualHardDrivescreen:

Acceptthedefault"Dynamicallyallocated"optionbyclickingthe"Next"button:

Acceptthedefaultfilenameandsetsizeto"20G",thenclickthe"Create"button:

The"rac1"VMwillappearonthelefthandpane.Clickonthe"Network"linkontherightside:

Makesure"Adapter1"isenabled,attachedto"InternalNetwork"or"HostonlyAdapter".Thisinetrface
willbeusedforpublicnetwork,forexample,forconnectiontotheRACdatbasefromotherapplications.
Moreaboutnetworkingwillbeexplainedlater.Onthescreenshotbelow"InternalNetwork"isselected
andname"pubnet"wasgiventothisnetwork:

Thenclickonthe"Adapter2"tab.Makesure"Adapter2"isenabledandattachto"InternalNetwork".
Namethisnetwork"privnet":

Thenfinallyclickonthe"Adapter3",enableit,andattachto"BridgedAdapter"or"NAT".Thisadapter
willbeusedforinternet.Thenpress"OK"button:

Optionally,youcandisabletheaudiocardusing"Audio"link.Thiswillprobablysavesomeamountof
spaceandavoidpotentialproblemsrelatedtoaudiosettings.Alsoifyoursystemhas4CPUcoresor
more,itwillmakesensetoallocate2CPUstotheVirtualMachine.Youcandothatin"System"
settings.
Thevirtualmachineisnowconfiguredsowecanstarttheguestoperatingsysteminstallation.

GuestOperatingSystemInstallation
PleasenotethatduringinstallationVirtualBoxwillbekeepingthemousepointerinsideVMarea.To
exit,pressRightControlkeyonthekeyboard.
WewilluseOracleLinux6.4ISOimagedownloadedearlier.Placethe.isofileintothevirtualDVD.
Select"Storage"linkontherighthandpaneoftheVirtualBoxManagerscreentoopen"Storage"
screen.ThenselectDVDdriveinthe"StorageTree"section:

In"Attributes"sectionclickontheDVDdiskiconandchooseDVD.isofile.Notethatnameofthefile
showsintheStorageTree.Thenpress'OK":

Startthevirtualmachinebyclickingthe"Start"buttononthetoolbar.Theresultingconsolewindowwill
containtheOracleLinuxbootscreen.Proceedwiththe"Installorupgradeanexistingsystem":

Donotperformthemediatest.Choose"Skip"button:

ContinuethroughtheOracleLinuxinstallationasyouwouldforanormalserver.Onnextthreescreens
selectLanguage,Keyboard,andBasicStorageDevicestype.Confirmtodiscardanydata.
Note.Forthissandboxsetupweuseddomainname"localdomain".Youcanchangeit,butkeepin
mindtomakesamechangethroughoutthisexercise.
Set"Hostname"torac1.localdomainandpress"ConfigureNetwork":

IntheNetworkConnectionsscreenselect"Systemeth0"interface,whichwillbeusedforpublic
network,andpress"Edit":

Makesurethat"Connectautomatically"ischecked.Select"IPv4Settings"tabchangeMethodto
"Manual"Press"Add"andfillAddress:192.168.56.71Netmask:255.255.255.0Gateway:0.0.0.0.
Press"Apply"thendone:

IntheNetworkConnectionsscreenselect"Systemeth1"interface,thiswillbeusedforprivatenetwork,
thenpress"Edit".Thencheckthebox"Connectautomatically".Select"IPv4Settings"tabchange
Methodto"Manual".Press"Add"andfillAddress:192.168.10.1Netmask:255.255.255.0Gateway:
0.0.0.0.Whendone,press"Apply":

Finallyselect"Systemeth2"interface,thiswillbeusedforInternet,thenpress"Edit".Checkthebox
"Connectautomatically".Select"IPv4Settings"tabmakesuretheMethodissetto"Automatic(DHCP)".
Press"Apply"button:

CloseNetworkConnectionsscreenandproceedtonextsetupscreen.SelecttimezoneTypeinRoot
Password:oracle
Select"UseAllSpace"typeofinstallationandcheck"Reviewandmodifypartitioninglayout":

Editsizeoflv_swapdeviceto1700MBtheneditsizeoflv_rootto18276MB.Press"Next":

Confirmthroughwarningsandcreatepartitions.KeepdefaultsinBootloaderscreen.
Inthesoftwaretypeinstallationscreenselect"DatabaseServer"andcheck"Customizenow"button.
PressNext:

IntheCustomizationscreenselect"Databases"anduncheckallitemsselect"Desktops"andcheck
"Desktop"and"GraphicalAdministrationTools"thenpressNextandfinishinstallation.Reboot.
Whenitcomesback,therewillbemoresetupscreensobvioustohandle.Don'tcreate'oracle'account,
thiswillbedonelater.YoucanskipKdumpsettings.Congratulations!TheLinuxhasbeeninstalled.
Tip.WewillneedtotouseTerminalprogram(MainMenu|Applications|SystemTools|Terminal)very
often.YoucansavetimebyaddingTerminalicontothePanelorDesktop.RightclickonTerminal
menuitemandselecteither"Addthislaunchertopanel"or"Addthislaunchertodesktop".

CheckInternetAccess
WewillneedInternetaccessbecauseadditionalpackageswillbeinstalledonline.Openterminal(Main
menu|Applications|SystemTools|Terminal)andtrytopinganyInternetsite,forexample:

pingyahoo.com

Ifpingdoesn'twork,troubleshoottheproblemusing'ifconfig'commandandmakingchangesin
NetworkConnections(LinuxdesktopMainmenu|System|Preferences|NetworkConnections).Ifyou
madechangesinNetworkConnections,restartinterfacebyrebootingVMorrunningthesetwo
commands:

#ifdowneth2
#ifupeth2

Thencheckthepingagain.

OracleClusterwareInstallationPrerequisites.
Part1
Allactionsinthissectionmustbeperformedbytherootuser.

Install'oraclerdbmsserver12cR1preinstall'package.Thispackageperformsprerequisitesincluding
kernelparameterchangeandcreationofLinuxoracleaccount:

#yuminstalloraclerdbmsserver12cR1preinstall

Note.Probablyyouwillnotbeabletopasteandcopythiscommand.Soyouwillhavetotypeit
manually.WearegoingtofixthatshortlybyinstallingGuestAdditions.Fornowjusttypethose
commands.
AnotherNote.Whileusing'yum'sometimesIhaveseenthiserror:"Existinglock/var/run/yum.pid:
anothercopyisrunningaspid523.Anotherappiscurrentlyholdingtheyumlockwaitingforitto
exit...".Whenthishappened,Iwasinterruptingthiswaitwith'CtrlC'andthenkillingreportedprocess,
inthisexample523:'kill523'.Afterthat,nextrunofyumwasalwaysOK.
TherearetwopreferredwaystomakeshareddrivesvisibletoASM:UdevandASMLib.Youhaveto
choosebetweenthem.Thereareprosandconsofoneoranotherway,youcanfinddiscussionson
thistopicinInternet.Ifyoudon'tcareandonlyinterestedinhavingRACinstalled,chooseASMLib
becauseitiseasieranduseslessofOSresources,whichismoresuitableforthelaptopsetup.Next
twoactionsshouldbedoneifASMLibisselected.SkipthemifyoudecideonUdevway.
1.InstallASMLib:

#yuminstalloracleasm
#yuminstalloracleasmsupport

2.ConfigureASMLibrunningthiscommandandansweringquestions:

#oracleasmconfigurei
ConfiguringtheOracleASMlibrarydriver.

ThiswillconfiguretheonbootpropertiesoftheOracleASMlibrary
driver.Thefollowingquestionswilldeterminewhetherthedriveris
loadedonbootandwhatpermissionsitwillhave.Thecurrentvalues
willbeshowninbrackets('[]').Hitting<ENTER>withouttypingan
answerwillkeepthatcurrentvalue.CtrlCwillabort.

Defaultusertoownthedriverinterface[]:oracle
Defaultgrouptoownthedriverinterface[]:oinstall
StartOracleASMlibrarydriveronboot(y/n)[n]:y
ScanforOracleASMdisksonboot(y/n)[y]:
WritingOracleASMlibrarydriverconfiguration:done
#

Tip.Wewillneedtoeditconfigurationfiles.Ifyouarenotfamiliarwith"vi"editororpreferGUIeditor,
install"gedit":

yuminstallgedit

Forexample,toedit"/etc/hosts"file,usethiscommand:"gedit/etc/hosts".

InstallGuestAdditions
GuestAdditionsareoptional,buthighlyrecommended.GuestAdditionsallowbetterintegrationof
mousesupportandclipboardbidirectionalcopying.Anotherimportantfeatureissupportofshared
foldersmakingfilesinHostOSvisibletoGuest.TheremainingofthisdocumentassumesthatGuest
Additionsareinstalled.
InordertoinstallGuestAdditions,rebootjustcreatedVM:Menu|System|ShutDown.Loginasroot.
TheninthewindowmenuselectDevices|InstallGuestAdditions.Gothroughthedownloaduntilyou
seeDVDAutorunscreen:

Press"OK",then"Run"tostartinstallation.
Note.Theinstallationcanfailcomplainingonmissingkerneluekdevelpackageprovidinga'yum'
commandtoinstallthispackage.Runthiscommandthat'swhyweneedInternetaccess.Alsoinstall
anotherpackage:'yuminstallgcc'.ThenreinstallGuestAdditionsbydoubleclickingon
VBOXADDITIONSDVDicononthedesktop,andclicking"OpenAutorunPrompt"button.
ShutdownVM.InVirtualBoxManagerwindowselect"General","Advanced"tab,andmakesurethat
"SharedClipboard"issetto"Bidirectional".Restartthemachine.Nowyoushouldbemuchhappier
aboutVirtualBox!

OracleClusterwareInstallationPrerequisites.
Part2
CreatethedirectoryinwhichtheOraclesoftwarewillbeinstalled.

mkdirp/u01/app/12.1.0/grid_1
mkdirp/u01/app/oracle
chownRoracle:oinstall/u01
chownoracle:oinstall/u01/app/oracle
chmodR775/u01/

Addoracleaccounttovboxsfgroups.ThevboxsfgroupwascreatedbyVirtualBoxGuestAdditionsand
willalloworacleuseraccessfoldersintheHostOS:

#usermodGdba,vboxsforacle

Resetoracleuserpasswordtooracle:

#passwdoracle
Changingpasswordforuseroracle.
Newpassword:
BADPASSWORD:itisbasedonadictionaryword
BADPASSWORD:istoosimple
Retypenewpassword:
passwd:allauthenticationtokensupdatedsuccessfully.
#

Disablesecurelinuxbyeditingthe"/etc/selinux/config"file,makingsuretheSELINUXflagissetas
follows.

SELINUX=disabled

EitherconfigureNTP,ormakesureitisnotconfiguredsotheOracleClusterTimeSynchronization
Service(ctssd)cansynchronizethetimesoftheRACnodes.InthiscasewewilldeconfigureNTP.

#servicentpdstop
Shuttingdownntpd:[FAILED]
#chkconfigntpdoff
#mv/etc/ntp.conf/etc/ntp.conf.orig
#rm/var/run/ntpd.pid

CleanupYUMrepositories:

#yumcleanall

Checkfilesystemusage,about2.8GBisused:

#dfk
Filesystem1KblocksUsedAvailableUse%Mountedon
/dev/mapper/vg_rac1lv_root
1842048427687241471603216%/
tmpfs154458410015444841%/dev/shm
/dev/sda14958447832239192217%/boot
#

NetworkSetup
Allactionsinthissectionmustbeperformedbytherootuser.
BelowisTCPlayoutofaddressesusedinpublicandprivatenetworks.Ifyouneedtouseanother
addresses,makecorrespondingadjustmentsandremembertostayconsistentwiththoseadjustments
throughouttherestofthearticle.Pleasenotethatthesubnet192.168.56.0isadefaultconfiguration
usedbytheVirtualBoxasHostonlynetworkconnectingthehostOSandvirtualmachines.The
VirtualBoxisalsorunningDHCPserveronthissubnetreservingaddressrange100254.Soitissafeto
useaddressesbelow100forstaticaddresses.Youcanverifythesesettingsin:Mainmenu|File|
Preferences|Network,thencheckthepropertiesoftheHostonlynetwork.Weareusingthissubnet
fortheRACpublicnetwork.Evenifyoudon'tneedtoconnectfromtheHostOStotheRAC,andyou
usedVB"InternalNetwork"forAdapter1,youstillcanuseproposedlayoutwithoutmakingchanges.
Edit"/etc/hosts"filebyappendingthefollowinginformation:

#Private
192.168.10.1rac1priv.localdomainrac1priv
192.168.10.2rac2priv.localdomainrac2priv

#Public
192.168.56.71rac1.localdomainrac1
192.168.56.72rac2.localdomainrac2

#Virtual
192.168.56.81rac1vip.localdomainrac1vip
192.168.56.82rac2vip.localdomainrac2vip

#SCAN
192.168.56.91racscan.localdomainracscan
192.168.56.92racscan.localdomainracscan
192.168.56.93racscan.localdomainracscan

Note.TheSCANaddressshouldnotreallybedefinedinthehostsfile.Insteadisshouldbedefinedon
theDNStoroundrobinbetween3addressesonthesamesubnetasthepublicIPs.Forthis
installation,wewillcompromiseandusethehostsfile.IfyouareusingDNS,thencommentoutlines
withSCANaddresses.
WealreadysetIPaddressesofalladaptersduringLinuxunstallation.Ifyoufollowedtheinstructions,
thereisnoneedtochangeanything.Butifyouneedtochangesomething,youcandosowiththe
NetworkConnectionstool:LinuxdesktopMainmenu|System|Preferences|NetworkConnections.
Nowweneedtodisablethefirewall:LinuxMainmenu|System|Administration|Firewall.Clickon
"Disable"icon,thenon"Apply".

DownloadedOracleInstallationFiles
TherearetwooptionstohandleOracledownloads:
DownloadortransferfilesintoVManduncompresstheminVM
DownloadanduncompressintheHostOS,thenmakefoldersaccessibletoVMfilesystem
Obviouslysecondoptionismuchbetterbecauseitdoesn'tusevirtualdiskofGuestVMandwillresult
insmallerfinalimage.Alsoinstallationfilescanbeeasilyreusedinanotherinstallationexercise.Inthis
sectionwearegoingtosetupVirtualBoxSharedFolders.
Itisassumedthatyoualreadydownloadedoracleinstallationfilesanduncompressedthemintothe
"grid"and"database"folders.Inourexamplethesefoldersarein"C:\TEMP\oracle_sw"folder.

C:\TEMP\oracle_sw>dirl
total0
drwx+1sromanenkoDomainUsers0Aug518:10database
drwx+1sromanenkoDomainUsers0Aug503:08grid

ShutdownVM.InVirtualBoxManagerclickon"SharedFolders"linkintherighthandpane.Addshared
folderbypressing"plus"icon.Thenselectpathtothelocationoforaclesoftware,andcheckboth
boxes"Readonly"and"Automount":

NoteYoucanuseanynamein"FolderName".Ifyouhaveoracleinstallationfilesatdifferentlocation,
youcanoverwritethatnameto"oracle_sw".Thiswillmakeeasiertofollowstepsbelow.
Press"OK"tosavethissetting.NowSharedFoldersshouldlooklikethis:

RestartVMandloginasoracleuser.Changedirectoryto"/media/sf_oracle_sw"thisiswhere
VirtualBoxmapsHostOSsharedfolder.NotethatVirtualBoxaddedprefix"sf_"tothenameofthe
folder.List'ls'contentofthefolder:

$cd/media/sf_oracle_sw
$ls
databasegrid
$

Note.Ifyouusedfoldernameotherthan"oracle_sw"whilecreatingsharedfolder,thenyouwillsee
"/media/sf_<foldername>"directory,pleasekeepthisinmindandmakecorrespondingadjustments.
Thereisonepackage'cvuqdisk'thatshouldbeinstalledbeforetheinstallation.InstallitfromtheOracle
grid/rpmdirectoryasrootuser:

$suroot
Password:
#cd/media/sf_oracle_sw/grid/rpm
#CVUQDISK_GRP=oinstall;exportCVUQDISK_GRP
#rpmUvhcvuqdisk*

ClonetheVirtualMachine
ShutdowntheVM.
IntheVirtualBoxManagerwindowstartclonewizard:Mainmenu|Machine|Clone.Type"rac2"forthe
nameofnewmachine.Makesurethat"ReinitializetheMACaddressofallnetworkcards"isnot
checked.Thenpress"Next":

Keepdefault"FullClone"optionselectedandpress"Clone":

StartclonedVMrac2andloginasrootuser.Thenchangehostnamebyeditingfile
"/etc/sysconfig/network",HOSTNAMEparameter:

HOSTNAME=rac2.localdomain

Start"NetworkConnections"tool(Mainmenu|System|Preferences|NetworkConnections).Editeth0
andeth1interfacesandsetinIPv4addresses192.168.56.72and192.168.10.2correspondingly.
Rebootsystem.
NowweneedtochangeMACaddressforallthreeinterfaces.AtthemomentwehavetwoVMswith
thesamesetofMACaddresses.Wecanrunonemachineoranother,butnotbothofthematthe
sametimebecauseMACaddressmustbeunique.Nochangeswillbemadetorac1,wewillpickup
threenewunusedaddressesandsetthemforeth0,eth1,andeth2inrac2.Theeasiestwaytodothat
istochangejustlasttwocharactersoftheaddress.Wearegoingtochangethemto'00'.Ifthelasttwo
charactersarealready'00',thenchangetosomethingelse,'01',forexample.Justmakesurethat
theseaddressesdon'tcollidewiththeMACaddressesofrac1.Inrunningrac2node,open"Network
Connections"andeditMACaddressinthe"Wired"tab.ThescreenshotbelowshowswheretosetMAC
address.Don'tforgettochangeMACaddressesforallthreeinterfaces.Pleasenotethatyoursetupwill
haveadifferentsetofMACaddressesbecausetheyarerandomgeneratedbyVirtualBox.

WritedownthenewMACaddressesforallthreeinterfaces.Savenewsettingspressing"Apply"button,
thenshutdownthemachine.Aftershutdown,returntotheVirtualBoxManager,selectrac2VMandedit
"Network"settings.MakesamechangestotheMACaddresses.Don'tforgettochangeMAC
addressesforallthreeadapters.

Startbothmachinesandcheckthattheycanpingeachotherusingbothpublicandprivatenetwork.
Forexample,onrac1:

$pingrac2
$pingrac2priv

Ifyouhaveproblems,use'ifconfig'commandtochecktheconfiguration,thencorrecttheproblem
using"NetworkConnections"tool.

CreateSharedDisks
Shutdownbothvirtualmachines.Weneedtocreateanewvirtualdisk,changeitsattributeto
ShareableandaddtobothVMs.InthecurrentversionofVirtualBox,theonlywaytocreateanewdisk
intheGUIisthroughthe"Storage"pageinthevirtualmachine'ssettings.Selecteitherrac1orrac2

VM,thenclickon"Storage"link.Select"SATAController"andclickon"AddHardDisk"icon.Ifnotsure,
whichicontouse,sameactionisavailablethroughthepopupmenu,rightclickonthe"SATA
Controller"andselect"AddHardDisk".

Press"Createnewdisk":

AcceptthedefaultVDItypeandclickthe"Next"buttonontheVirtualDiskCreationWizardwelcome
screen:

Select"Fixedsize"optionandpressthe"Next"button:

Changethenameandlocationofthisdisk.Youcankeepthisfileinthedefaultlocationthefolderofa
selectedVM.Although,becausethisdiskisshared,itwillbebettertoputitintheparentdirectory.So,
insteadof"...\VirtualBoxVMs\rac1"directory,placeitin"...\VirtualBoxVMs".Setthesizeto
"4GB".Ifyouwillneedmorespace,youcanchoosethebiggersize.And,regardlessofwhatyou
decidenow,itwillbepossibletoaddmoreshareddiskstotheASMgroupaftereverythingisinstalled.

CreatethenewdiskandthisdiskwillbealreadyattachedtoVM.
Selectthisnewdisk.YouwillseeinthediskInformationthatthetypeofthisdiskis"Normal".Weneed
tochangethisto"Shareable".Therewasnooptioninthepreviousdialogwindowstocreatenewdisk
as"Shareable".PressOKandclosestoragesettings.IntheVirtualBoxManageropenVirtualMedia
Manager:(Mainmenu|File|VirtualMediaManager).SelectthisnewdiskintheVirtualMedia
Manager:

Clickon"Modify"iconandselect"Shareable":

AttachthisexistingdisktotheotherVMusing"Storage"page.Don'tforgettoselectcorrectcontroller
beforeattachingthediskanduse"Chooseexistingdisk"option.

Intheendthe"Storage"sectionofbothVMsshouldbelookinglikethis:

Starteitherofthemachinesandloginasroot.Thecurrentdiskscanbeseenbyissuingthefollowing
commands.

#ls/dev/sd*
/dev/sda/dev/sda1/dev/sda2/dev/sdb
#

Usethe"fdisk"commandtopartitionthenewdisk"sdb".

#fdisk/dev/sdb
DevicecontainsneitheravalidDOSpartitiontable,norSun,SGIorOSFdisklabel
BuildinganewDOSdisklabelwithdiskidentifier0xd724aa83.
Changeswillremaininmemoryonly,untilyoudecidetowritethem.
Afterthat,ofcourse,thepreviouscontentwon'tberecoverable.

Warning:invalidflag0x0000ofpartitiontable4willbecorrectedbyw(rite)

WARNING:DOScompatiblemodeisdeprecated.It'sstronglyrecommendedto
switchoffthemode(command'c')andchangedisplayunitsto
sectors(command'u').

Command(mforhelp):n
Commandaction
eextended
pprimarypartition(14)
p
Partitionnumber(14):1
Firstcylinder(1391,default1):
Usingdefaultvalue1
Lastcylinder,+cylindersor+size{K,M,G}(1391,default391):
Usingdefaultvalue391

Command(mforhelp):w
Thepartitiontablehasbeenaltered!

Callingioctl()torereadpartitiontable.
Syncingdisks.
#

Thesequenceofanswersis"n","p","1","Return","Return"and"w".
Oncethenewdiskispartitioned,theresultcanbeseenbyrepeatingtheprevious"ls"command.

#ls/dev/sd*
/dev/sda/dev/sda1/dev/sda2/dev/sdb/dev/sdb1
#

IfyoudecidedearlieronASMLibwayofexposingshareddiskstoASMthendo"CreateASMLibDisk"
section,otherwisedo"UdevConfiguration"below.

CreateASMLibDisk
MarkthenewshareddiskintheASMLibasfollows.

#oracleasmcreatediskDISK1/dev/sdb1
Writingdiskheader:done
Instantiatingdisk:done
#

Runthe"scandisks"commandtorefreshtheASMLibdiskconfiguration.

#oracleasmscandisks
Reloadingdiskpartitions:done
CleaninganystaleASMdisks...
ScanningsystemforASMdisks...
#

WecanseethediskisnowvisibletoASMusingthe"listdisks"command.

#oracleasmlistdisks
DISK1
#

StartanotherVMandloginasroot.CheckthattheshareddiskisvisibletoASMusingthe"listdisks"
command.

#oracleasmlistdisks
DISK1
#

Thevirtualmachinesandshareddisksarenowconfiguredforthegridinfrastructure!Optionally,you
cansavethisstateusingSimplifiedWayofSaving/RestoringImages.Pleaseskipthenextsection
"UdevConfiguration".

UdevConfiguration
Thisconfigurationneedstobedoneonbothnodes,youcandoitstepbystepinparallelonboth
nodes.
Open"/etc/scsi_id.config"file(forexample,"gedit/etc/scsi_id.config",ifthefiledoen'texist,itwillbe
created)andaddthisline:

options=g

Execute"scsi_id"filecommandfortheshareddisk.

#/sbin/scsi_idgud/dev/sdb
1ATA_VBOX_HARDDISK_VBb6f0fac5f473897a

YouwillhavedifferentSCSIIDoutput.Thisidwillbeusedinudevrulesfile.Edit"/etc/udev/rules.d/99
oracleasmdevices.rules"fileandaddfollowingentryasoneline.MakesureyouuseSCSIIDfromthe
previouscommand.

KERNEL=="sd?1",BUS=="scsi",PROGRAM=="/sbin/scsi_idgud/dev/$parent",RESULT=="1ATA_VBO

RestartVM,loginasoracleandcheckthatshareddiskcanbeexposedtoASM:

$lsal/dev/asm*
brwrw1oracledba8,17Oct613:31/dev/asmdisk1

Thevirtualmachinesandshareddisksarenowconfiguredforthegridinfrastructure!Optionally,you
cansavethisstateusingSimplifiedWayofSaving/RestoringImages.

InstalltheGridInfrastructure
Makesurethe"rac1"and"rac2"virtualmachinesarestarted,thenloginto"rac1"asoracleuser(better
toavoidusing"su"command,Ihadproblems)andstarttheOracleinstaller.

$cd/media/sf_oracle_sw/grid
$./runInstaller

Select"Skipsoftwareupdates"option,press"Next":

Selectthe"InstallandConfigureGridInfrastructureforaCluster"option,thenpressthe"Next"button.

Selectthe"ConfigueaStandardcluster"option,thenclickthe"Next"button.

Selectthe"AdvancedInstallation"option,thenclickthe"Next"button.

SelectLanguageandgotothenextscreen.Onthe"GridPlugandPlayinformation"screen,change
ClusterNameto"raccluster"andSCANNameto"racscan.localdomain",uncheck"ConfigureGNS"
box,thenpressthe"Next"button.

Onthe"ClusterNodeConfiguration"screen,clickthe"Add"button.

Enterthedetailsofthesecondnodeinthecluster,thenclickthe"OK"button.

Clickthe"SSHConnectivity..."buttonandenterthepasswordforthe"oracle"user.Clickthe"Setup"
buttontoconfigureSSHconnectivity,andthe"Test"buttontotestitonceitiscomplete.Thenpress
"Next".

Onthe"SpecifyNetworkInterfaceUsage"screencheckthePublicandPrivatenetworksarespecified
correctly.Pressthe"Next"button.

Onthe"GridInfrastructureManagementRepositoryOption"screencheck"No"andpressthe"Next"
button.Confirmthisdecisiononthenextscreen.

Onthe"StorageOptionInformation"screenkeepOracleASMoptionselectedandpress"Next".

Onthe"CreateASMDiskGroup"screenclickon"ChangeDiscoveryPath"button:

IfusearlierusedASMLibconfiguration,thenenter"/dev/oracleasm/disks"(showninthescreenshot
below).IfyouusedUdevconfiguration,thenenter"/dev/asm*".Press"OK":

Keep"DiskGroupName"unchanged.Select"External"redundancyoption.Check
"/dev/oracleasm/disks/DISK1"or"/dev/asmdisk1"inthe"AddDisks"section.Whendone,press"Next".

Onthe"SpecifyASMPassword"screenselect"Usesamepasswordsfortheseaccounts"optionand
typein"oracle"password,thenpress"Next".Ignorewarningsaboutpasswordweakness.

Keepdefaultsonthe"FailureIsolationSupport"andpress"Next".

Keepdefaultsonthe"PrivilegedOperatingSystemGroups"andpress"Next".Ignorewarningonthe
nextscreen.

KeepsuggestedOraclebaseunchanged,andadjustSoftwarelocationonthe"SpecifyInstallation
Location"andpress"Next".

Keepsuggestedpathunchangedonthe"CreateInventory"andpress"Next".

Inthe"Rootscriptexecutionconfiguration"select"Automaticallyrunconfigurationscripts"andcheck
"Userootoption",thentypepasswordoracle.Press"Next".

Theresultsofprerequisitechecksareshownonthenextscreen.Youshouldseethreewarningsand
onefailure.ThefailurewascausedbyinabilitytolookupSCANinDNSandthatshouldbeexpected.
Check"IgnoreAll"boxandpress"Next".

Press"Install"ontheSummaryscreen.

Waitwhileinstallationtakesplace...Whenaskedtoconfirmrootscriptsexecution,answerYes.The
"rootscript"stepwasthelongestone,sobepatient!

WeexpecttheverificationphasetofailwitherrorsrelatedtolowHostRAM,SwapFilesizes,group
mismatchforDISK1,and"udevinformation".Theseerrorscanbeignored.Youcancheckerrors
openinglogfile.Forexample,forthelogfileshownbelow"gedit
/u01/app/oraInventory/logs/installActions20130930_102544PM.log",thensearchfor"Error".

Hereistheexampleofoffendinglinesfromthelogfileforrac1node:

INFO:INFO:ErrorMessage:PRVF7530:Sufficientphysicalmemoryisnotavailableonnode"rac
INFO:INFO:Cause:Amountofphysicalmemory(RAM)founddoesnotmeetminimummemoryrequirem
INFO:INFO:Action:Addphysicalmemory(RAM)tothenodespecified.

...

INFO:INFO:ErrorMessage:PRVF7573:Sufficientswapsizeisnotavailableonnode"rac1"[Re
INFO:INFO:Cause:Theswapsizefounddoesnotmeettheminimumrequirement.
INFO:INFO:Action:Increaseswapsizetoatleastmeettheminimumswapspacerequirement.

...

INFO:INFO:ERROR:[Result.addErrorDescription:618]PRVF9992:Groupofdevice"/dev/oraclea

...

oracle
INFO:INFO:ERROR:[Result.addErrorDescription:607]PRVF9802:Attempttogetudevinformati
INFO:NoUDEVrulefoundfordevice(s)specified
INFO:

Providedthesearetheonlyerrors,itissafetoignorethemandcontinuebyclickingthe"Next"button.
ClosetheConfigurationAssistantonthenextscreen.
Checkthestatusofrunningclusterware.Onrac1asrootuser:

#.oraenv
ORACLE_SID=[oracle]?+ASM1
TheOraclebasehasbeensetto/u01/app/oracle

#crsctlstatusresourcet

NameTargetStateServerStatedetails

LocalResources

ora.DATA.dg
ONLINEONLINErac1STABLE
ONLINEONLINErac2STABLE
ora.LISTENER.lsnr
ONLINEONLINErac1STABLE
ONLINEONLINErac2STABLE
ora.asm

ONLINEONLINErac1Started,STABLE
ONLINEONLINErac2Started,STABLE
ora.net1.network
ONLINEONLINErac1STABLE
ONLINEONLINErac2STABLE
ora.ons
ONLINEONLINErac1STABLE
ONLINEONLINErac2STABLE

ClusterResources

ora.LISTENER_SCAN1.lsnr
1ONLINEONLINErac2STABLE
ora.LISTENER_SCAN2.lsnr
1ONLINEONLINErac1STABLE
ora.LISTENER_SCAN3.lsnr
1ONLINEONLINErac1STABLE
ora.cvu
1ONLINEONLINErac1STABLE
ora.oc4j
1OFFLINEOFFLINESTABLE
ora.rac1.vip
1ONLINEONLINErac1STABLE
ora.rac2.vip
1ONLINEONLINErac2STABLE
ora.scan1.vip
1ONLINEONLINErac2STABLE
ora.scan2.vip
1ONLINEONLINErac1STABLE
ora.scan3.vip
1ONLINEONLINErac1STABLE

Youshouldseevariousclusterwarecomponentsrunningonbothnodes.Iftherewereserious
problemsduringinstallation,thecomponentsofclusterwarewouldn'tstart.Alsoyoucancheck
clusterwarealertfile"gedit/u01/app/12.1.0/grid_1/log/rac1/alertrac1.log"(onrac1node,make
adjustmentsforrac2).
Thegridinfrastructureinstallationisnowcomplete!Optionally,youcansavethisstateusingSimplified
WayofSaving/RestoringImages.
Checkfilesystemusage,about8.6GBareused:

$dfk
Filesystem1KblocksUsedAvailableUse%Mountedon
/dev/mapper/vg_rac1lv_root
184204848691564879319250%/
tmpfs154458464974089484443%/dev/shm
/dev/sda14958447832239192217%/boot$

InstalltheDatabaseSoftware
Makesurethe"rac1"and"rac2"virtualmachinesarestarted,thenloginto"rac1"orswitchtheuserto
oracleandstarttheOracleinstaller.

$cd/media/sf_oracle_sw/database
$./runInstaller

Uncheckthe"Iwishtoreceivesecurityupdates..."checkboxandpressthe"Next"button:

Check"Skipsoftwareupdates"checkboxandpressthe"Next"button:

Check"Installdatabasesoftwareonly"optionandpress"Next"button:

Select"OracleRealApplicationClusterdatabaseinstallation"andthenpress"Next"button.

Makesurebothnodesareselected.Presson"SSHConnectivity"enter"oracle"passwordandthen
press"Setup"button.AftertheSSHconnectivitysetupisfinished,press"Test"buttontotestit.Then
pressNextbutton.

SelectLanguageonnextscreenandpressthe"Next"button.
Accept"EnterpriseEdition"optionandpressthe"Next"button:

Acceptdefaultinstallationlocationsandpressthe"Next"button:

Acceptdefaultgroupsin"PrivilegedOperationSystemgroups"andpressthe"Next"button:

Waitfortheprerequisitechecktocomplete.Ifthereareanyproblems,eitherfixthem,orcheckthe
"IgnoreAll"checkbox.Onthescreenshotbelow3warningsareshownwhicharesafetoignore.Click
the"Next"button.

Ifyouarehappywiththesummaryinformation,clickthe"Install"button.

Waitwhiletheinstallationtakesplace.

Executerootscriptswhenpromptedonbothnodes.

#/u01/app/oracle/product/12.1.0/dbhome_1/root.sh
PerformingrootuseroperationforOracle12c

Thefollowingenvironmentvariablesaresetas:
ORACLE_OWNER=oracle
ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1

Enterthefullpathnameofthelocalbindirectory:[/usr/local/bin]:
Thecontentsof"dbhome"havenotchanged.Noneedtooverwrite.
Thecontentsof"oraenv"havenotchanged.Noneedtooverwrite.
Thecontentsof"coraenv"havenotchanged.Noneedtooverwrite.

Entrieswillbeaddedtothe/etc/oratabfileasneededby
DatabaseConfigurationAssistantwhenadatabaseiscreated
Finishedrunninggenericpartofrootscript.
Nowproductspecificrootactionswillbeperformed.
#

Press"Close"buttonandtheOracleDBsoftwareisnowinstalled!Optionally,youcansavethisstate
usingSimplifiedWayofSaving/RestoringImages.

InstalltheDatabase
StartdbcaDBConfigurationAssistant:

$cd/u01/app/oracle/product/12.1.0/dbhome_1/bin
$./dbca

Accept"CreateDatabase"optionandpressthe"Next"button:

Check"Createadatabasewithdefaultconfiguration"typein"GlobalDtabaseName"thedesired
namewehaveselectedAL32UTF8characterset,makeyourchoiceasneededalsosupplya
passwordforSYS.Wearegoingtocreateareqular(pre12ctype)database,the"CreateAsContainer
Databaseisunchecked.PressNextbutton:

Waitfortheprerequisitechecktocomplete.Theonlyreportedproblemwasallfamilar"SwapSize".
Check"IgnoreAll"checkboxandclickthe"Next"button.

Press"Finish"buttonintheSummaryscreen:

Clickthe"Close"buttontoexittheinstaller.

TheRACdatabasecreationisnowcomplete!

ChecktheStatusoftheRAC
ThereareseveralwaystocheckthestatusoftheRAC.The srvctl utilityshowsthecurrent
configurationandstatusoftheRACdatabase.

$.oraenv
ORACLE_SID=[oracle]?racdb
TheOraclebasehasbeensetto/u01/app/oracle

$srvctlconfigdatabasedracdb
Databaseuniquename:racdb
Databasename:racdb
Oraclehome:/u01/app/oracle/product/12.1.0/dbhome_1
Oracleuser:oracle
Spfile:+DATA/racdb/spfileracdb.ora
Passwordfile:+DATA/racdb/orapwracdb
Domain:localdomain
Startoptions:open
Stopoptions:immediate
Databaserole:PRIMARY
Managementpolicy:AUTOMATIC
Serverpools:racdb
Databaseinstances:racdb1,racdb2
DiskGroups:DATA
Mountpointpaths:
Services:
Type:RAC
Startconcurrency:
Stopconcurrency:
Databaseisadministratormanaged

$srvctlstatusdatabasedracdb
Instanceracdb1isrunningonnoderac1
Instanceracdb2isrunningonnoderac2
$

The V$ACTIVE_INSTANCES viewcanalsodisplaythecurrentstatusoftheinstances.

$exportORACLE_SID=racdb1
[oracle@rac1Desktop]$sqlplus/assysdba
SELECTinst_nameFROMv$active_instances;

INST_NAME

rac1.localdomain:racdb1
rac2.localdomain:racdb2

exit
$

MakingImagesoftheRACDatabase
Atanypointearlierwecouldsavetheimageofcreatedvirtualmachineandthenrestoreitatwill.Here
wearegoingtosaveimagesofthenewlycreatedOracleRACsystemwhichwecanrestoreinthe
samesystemorevenhandovertoanotherlocationandrestoreinamatteroffewminutes!
IfallyouneedissavingandrestoringimagesinthesameVirtualBoxsetup,youcanuseSimplifiedWay
ofSaving/RestoringImages.
TheexportofVMisastraightforwardprocessandsavingRACimageswouldbeaneasytaskifnot
dealingwiththeshareddisk.Inmyviewthesimplestwaytohandlethatisbydetachingshareddisk
frombothnodesandtakingcareofthesethreeparts(twoselfcontainedVMsandoneShareddisk)
separately.Intheendtherewillbethreefiles:twofilesforVMsandafilerepresentingtheshareddisk.
Thesethreefilescanbefurtherzippedbyyourfavoritearchiverintoonefilewhichcanbeusedfor
storageortransfer.Afterexportisdone,theshareddiskcanbeeasilyattachedbacktothenodes.
SameistruefortheimportofVMsbackintoVirtualBoxalongwiththecopyofshareddisktheshared
diskisattachedtotheimportedVMsasanextrastep.Let'sperformalltheseactions.

CleanShutdownofRAC
Butfirst,weneedtoshutdownserversinniceandcleanmannerbecausewewantsavethemina
robuststate.Shutdownthedatabase.Asoracleuserexecuteonanynode:

$.oraenv
ORACLE_SID=[oracle]?racdb
TheOraclebasehasbeensetto/u01/app/oracle

$srvctlstopdatabasedracdb
$

Shutdowntheclusterwareonthefirstnode.Asrootuserexecute:

#.oraenv
ORACLE_SID=[racdb1]?+ASM1
TheOraclebaseremainsunchangedwithvalue/u01/app/oracle

#crsctlstopcrs
...
CRS4133:OracleHighAvailabilityServiceshasbeenstopped.
#

Shutdowntheclusterwareonthesecondnode.Asrootuserexecute:

#.oraenv
ORACLE_SID=[racdb1]?+ASM2
TheOraclebaseremainsunchangedwithvalue/u01/app/oracle

#crsctlstopcrs
...
CRS4133:OracleHighAvailabilityServiceshasbeenstopped.
#

Shutdownbothvirtualmachines.WaituntilallVMwindowsareclosed.

DetachSharedDiskandMakeaCopyOfIt
IntheVirtualBoxManageropenVirtualMediaManager:Mainmenu|File|VirtualMediaManager.
ThenselectthediskusedbytheRAC(rac_shared_disk1.vdi).Notethatthisdiskshowsasattachedto
rac1andrac2VMs:

Clickon"Release"iconandthenconfirminthepopupwindow.Notethatthisdisknowshowsas"Not
attached".Clickon"Copy"tostartDiskCopyingWizard.

AcceptVirtualdisktocopyandpress"Next".

AcceptVirtualdiskfiletypeasVDIandpress"Next".

Select"Fixedsize"andpress"Next".

Onthenextscreenyoucansetlocationandnameofthenewfile.Whendone,press"Next".

OntheSummaryscreenreviewdetailsandpress"Copy"tocompletecopying.ClosetheMedia
Managerwhencopyingisdone.
Note.Donottrytocopy.vdifilebecausethecopywillretainsamediskUIDandVirtualBoxwillrefuse
touseitbecausethereisalreadysuchdisk.WhencopyingtroughtheVirtualMediaManager,thenew
UIDisassignedautomatically.

ExportVMs
IntheVirtualBoxManagerselectVM,thencallApplianceExportWizard:Mainmenu|File|Export
Appliance.Exportingisgenerallyassimpleassavingafile.ExportbothVMs.
Nowyoushouldhave3filesthatcanbefurtherzippedintoasinglefilewiththesizeabout12GB.

ReattachSharedDisktotheOriginalRACSetup
FixourcurrentworkingRACsetupbyreattachingshareddisktorac1andrac2VMusing"Storage"
page.Don'tforgettoselectcorrectcontrollerbeforeattachingthedisk:

Press"AddHardDisk"iconanduse"ChooseExistingDisk"toattach rac_shared_disk1.vdi .Once

ShareddiskisattachedtobothVMs,theRACisreadytorun.

RestoringRACfromSavedFiles
InthissectionwewillimportRACfromthesavedfilescreatingasecondRACsystem.Don'trunboth
RACatthesametimebecausetheywillhavesamenetworkattributes.
OpenApplianceImportWizard:Mainmenu|File|ImportAppliance.Choosethefileandpress"Next":

OntheApplianceImportSettingsdifferentattributesofnewVMcanbechanged.Wearegoingto
acceptsettingsunchanged.Itisinterestingtonote,thatdisksaregoingtobeimportedinVMDKformat
differentfromtheoriginalVDIformat.

WaituntiltheVMisimported:

ImportbothVMsandcopySharedDisk rac_shared_disk1_copy.vdi fileintotheparentdirectory

(VirtualVMs).Thisdiskcouldbeattachedtobothmachines,butunfortunatelycurrentversion(4.1.18)
ofVirtualBoxdoesn'tpreservetypeofthediskthenmakingacopy.Attachthisdisktotheeitherof
importedVM,thenselectitandreviewdiskinformation:

IntheVirtualBox4.1.18,thecopieddiskhas"Normal"type.Ifyouhaveanewerversionandthetypeis
"Shareable"thenthisbughasbeenfixed,andyoucanproceedtoanotherVM.Ifnot,deattachthe
disk,thengototheVirtualMediaManagerandchangethedisktypeto"Shareable"ashasbeen
describedabove,thenreturntotheVirtualmachinesandattachedtheShareddisk.
StartnewVMs.Theclusterwareshouldstartautomatically.Youwillneedtobringupthedatabase.
Loginastheoracleuserandexecute:

$.oraenv
ORACLE_SID=[oracle]?racdb
TheOraclebasehasbeensetto/u01/app/oracle

$srvctlstartdatabasedracdb
$

TheRACshouldbewellandrunning!

SimplifiedWayofSaving/RestoringImages
Thisisnotasupportedwaytosave/restoreimagesinVirtualBox,butitisverysimpleandworkedwell.
TherestoreislimitedtotheVirtualBoxsetupwhereimagesweresaved.
StopbothVMs.FindthelocationwhereVBstoresimages:Mainmenu|File|Preferences|General,
"DefaultMachineFolder"box.Forexample:"C:\Users\sromanenko\VirtualBoxVMs".Gotothisfolder
andlocate"rac1","rac2"directoriesand"rac_shared_disk1.vdi"file.Makea.zipor.rarfilecontaining
thesethreeitems.Ifyoudon'thavecompressingtoolsandthereisenoughdiskspace,youcancopy
thesethreeitemstoanotherlocation.Theimagesaresaved!
Inordertorestoretheseimages:
MakesureVMsarestopped
Gotothe"VirtualBoxVMs"folderanddelete"rac1","rac2"directories,and
"rac_shared_disk1.vdi"file
Restorethesethreeitemsfromthe.zipfileorcopyfromthelocationwheretheseitemswere
saved
TherestoredVMsarereadytorun.
That'sit!Pleasesendyourfeedbackto:
BacktotheTop.

Comments
CliffordMathewsays:
January1,2014at20:24,GMT
Thankyoufortheinstructions.
DuringtheGRIDinstallation,IencounterederrorPRVF7617on"SettingUpGridInfrastructure
Step8of19".Essentially,themulticasttestswerefailing.Toidentifyand"workaround"(see
CAUTIONbelow)thiserror,thefollowingstepsweretaken:
1.Executedthe"runcluvfy"commandasshownbelowfromtheshellprompt.Thiscommandfile
islocatedinthesamedirectoryasthe"runInstaller".
./runcluvfy.shstageprecrsinstnrac1,rac2verbose
2.Turnedofftheipv4firewallusingthefollowingcommandsonbothRACnodes(rac1andrac2)
serviceiptablesstop
chkconfigiptablesoff
3.RestartbothVMs,andreexecuteStep1abovetomakesureerrorsareallclearedup
4.ContinuewithGRIDinstallasisdocumentedinthisblog.

CAUTION:ByshuttingdowntheFirewallasIdid,Imadethesystemvulnerablefromasecurity
standpoint.Apropersolutionshouldinvolveconfiguringthe"iptables"correctlytoallowwhat
Oraclewants.
AtthispointIdon'thavethestepstodothis"proper"solution.
Notes:
======
IfanyonewishestodisableIPV6,thestepsbelowworkedforme.IPV6wasNOTthecausefor
theaboveissuessothisisnotarequiredstep.
vi/etc/hosts#andcommentoutreferencestoanyipv6addressesbuplacinga"#"characterin
front,andsave
vi/etc/sysconfig/network#andaddthelinesgivenbelow,andsave
NETWORKING_IPV6=no
IPV6INIT=no
vi/etc/modprobe.conf#andaddthelinegivenbelow,andsave(thisfilewasnotpresent,soI
createditbysaving)
installipv6/bin/true
chkconfigip6tablesoff
Restartthemachines

SergeiRomanenkosays:
January1,2014at23:52,GMT
Thankyouforthecomment.Letmeaskifyoualreadydisabledfirewall(asdescribedin
"NetworkSetup"section)beforegettingPRVF7617error?
CliffordMathewsays:
January2,2014at00:26,GMT
AsecondcluvfyfailureoccurredattheendofGRIDinstall(runningfromserverrac1).This
hadtodowithSCANaddressesandnslookup.ToresolvethisIinstalledBINDononeof
theservers(rac1)usinginstructionstheURLbelow:
http://www.oraclebase.com/articles/linux/dnsconfigurationforscan.php(Thispagerefers
toaGUIDNSeditorIcouldnotfindthatpackage,sothatwasnotinstalled.Allrequired
DNSfileswerecreatedwhen"servicenamedstart"commandwasissued.Afterstarting
"named",Ifollowedtheinstructionsandedited(orcreatedtheadditional)filesmentionedin
thatarticle.Ionlyaddedthe3SCANaddressesandlocalhostintheDNSdatabase.These
3SCANaddresseswerethenremovedfrom/etc/hostsbycommentingthemout.Dida
final"servicenamedrestart",thentried"nslookupracscan.localdomain"whichresolved

thenameto3IPaddresses.
Afterthisclicked"Retry"buttontheinstallerwindow,anditsucceeded.
CliffordMathewsays:
January2,2014at00:29,GMT
HelloSergei,Imighthavemissedthe"disablefirewall".Sorry.
SergeiRomanenkosays:
January2,2014at02:02,GMT
Noproblem.Itisgoodyoumentionedthatdisablingfirewallisnotabestthinginanyreal
worldsystem.
JohnHongsays:
January1,2014at21:47,GMT
Hi,forthosepeoplewhoishavingthisissuepleaseseethelink
TOinstallguessadditionerrorinthelogthatunabletofindthesourceofyourkernel...."
runkernelupdatereferthislinkhttp://oraclelabspace.blogspot.com/2012/06/settingupguest
additionsinoracle.html
SergeiRomanenkosays:
January2,2014at02:09,GMT
Yes,VirtualBoxguestadditioncanbetrickyandprobablyversiondependent.So,you
commentshouldbehelpful.Thanks!
Pragneshsays:
January9,2014at17:22,GMT
Hi,
Nicedocument.
IinstalledVMRAC2nodeandthenIwantedtoaddanode.WhenIclonethemachineandrun
clufyaftersuccessfullyinstallingtheGRIDsoftware,RAC3isnotshowingaspartoftheRAC
clusetr.DidImissanything?Istillwentaheadandinstalledthedatabasesoftwaresuccessfully.
ThenwhenIrantheDBCAfromRAC1tocreateinstanceonRAC3,afterenteringtheSYS
usernameandpasswordIgoterrormessage:serviceRAC1vip:1521:RAC1isnotavailable.
Checkyourusername,passwordandmakesurethelistenerisupandavailable"
Idon'tunderstandthiserror.Thescanandtheregularlistenerareupandrunning.DidIdo
anythingwrong?
Pleasehelp.
AkashShandilyasays:
January15,2014at20:31,GMT
GreatDocument!
Justanote:Igotanotherfailedprerequisite.
PRVE0426:Thesizeofinmemoryfilesystemmountedas/dev/shmis"____"megabytes

whichislessthantherequiredsizeof"2048"megabytesonnoderac1,rac2
Justincaseyougetthiserror,modifythesizeoftmpfsusingthefollowingsteps:
#vi/etc/fstab
Modifythetmpfslinetolooksomethinglike:
tmpfs/dev/shmtmpfssize=3g00
savethefileandquit
remounttmpfs:
mountoremounttmpfs
checkchanges:
dfh
Runprerequisitechecksagain.Solved.
Enjoy!
Costisays:
January25,2014at08:36,GMT
Hi.Ifollowedyourtutorial,stepbystepandeverytimeIhavedifferentspaceavai.Forexample
nowIhavealmostfree6GB,thereforeIwillhaveanerrorwheninstallingtheOracleDB.Doyou
knowwhy?Isitpossibletobeincreased,otherwiseIwillhavetotakeitagainallover.Thankyou
SergeiRomanenkosays:
January25,2014at19:37,GMT
Requireddiskspacecanfluctuateandeachnewinstallationcanbedifferent,Ididn'tkeep
exactnumbers,butitwasnotverybig.Thiscanbedependentonmanythingsincluding
Linuxversion,whichkeepschanging,yumonlinerepositorywhichalsogetsbiggerover
timeandyourVMwillcacheit.Bytheway,don'tforgettoreleaseyumrepositorywhen
youdon'tneeditanymore:'yumcleanall'.
Ithinkitispossibletoresize.VDIdiskusing'VBoxmanagemodifyhd'commandandthen
modifyingLinuxpartitionsby'GParted'butInevertriedthat.YoucansearchInternetfor
detailsonthat.
Prakashsays:
June10,2014at15:53,GMT
In"InstallGuestAdditions",SometimeGuestAdditionFailstoInstall.
ItwillbesafetoExecutebelowTwoCommandsFollowedby"Reboot"forSmoothadditionof
SharedFolder.
$yuminstallkernel*dkmsgcc

$yuminstallkerneluekdevel
ThisWillEnsureyour"GuestAdditions"installationalwaysSucceeds.
SergeiRomanenko(http://kab128.com)says:
June10,2014at16:15,GMT
Prakash,
Thankyouforyourcomment.VirtualBoxGuestAdditionisabittricky,soyourhintshould
help.
Jimsays:
June29,2014at01:14,GMT
Isuccessfullycreateatwonoderac12cusingoraclevirtualbox(nodeol6121rac1,ol6121
rac2)systemworksfine.
NowIwanttocreateathirdnode.Itriedusingvboxtocloneanewserverol6121rac3.itcloned
okandIresettheipaddressforthenewnode,.etc.
Sincethenode1andnode1areusingGridInfrastutueandASM,mynode3clonehaswhatnode
onehadinstall.HOwdoIchangenode3tobeadded.
ItalreadyhastheGRIDHomeandORACLEHOME...
Billsays:
July4,2014at11:14,GMT
Great.Icreaterac1andrac2.Howdoyouaddathirdnode.
Thanks
SergeiRomanenko(http://lab128.com)says:
July6,2014at02:44,GMT
Jim,Bill:
AlthoughIdidn'ttrythat,Iwoulddothesesteps:
removeonenode,sayrac1,fromthecluster
clonerac1torac3,doingallneededchanges(MACaddresses,hostname,IPaddresses
etc)describedinthearticle
addtwonewnodes(rac1,rac3)tothecluster.
RemovingandaddingnodesisastandardprocedurecoveredinOracledocumentation,
alsoavailableonmanywebsites.
Billsays:
July6,2014at22:19,GMT
Iwasabletocreate/addathirdnoderac3toyoursetupabove.Willsendthedocs.
Thanks
AnibalLopezsays:
August15,2014at09:16,GMT

HiSergei,
TheshareddrivesarenotmountedontheLinuxenvironment.
Isthereawaytocheckthis?
Doweneedtodothemountofthesharedstoragemanually?
Thx,
Anibal
SergeiRomanenko(http://www.lab128.com)says:
August15,2014at17:54,GMT
No.Checkthatshareddriveisaddedandaccessible:
#ls/dev/sd*
Theoutputshouldbesimilartothis:
/dev/sda/dev/sda1/dev/sda2/dev/sdb/dev/sdb1
AnibalLopezsays:
August15,2014at19:31,GMT
HiSergei,
TheotherproblemistheSharedFolder.
Ihaveaddedthedirectoryetc.
Whendoing:[root@rac1media]lsl
thereisnotoracle_sw.
YouknowifIcancheckthistoo?
Thx,
Anibal
SergeiRomanenko(http://www.lab128.com)says:
August15,2014at20:36,GMT
InVirtualBoxManager,selectVMandthenopenSharedFolders.Checkthat"oracle_sw"
isinthere.Ifnot,seethedocumentandaddit.YoumayneedtorestarttheVMifyou
madechanges.
Thenin"/media"youshouldseesf_oracle_swfolder.NotethatVBadds"sf_"prefix.
AnibalLopezsays:
August16,2014at13:20,GMT
HiSergei,
Ihavedonethisbutwithoutsuccess.

Isthereanywaytocheckwhy?
Isthereamissingoptionhere?
Thx,
Anibal
SergeiRomanenko(http://www.lab128.com)says:
August16,2014at14:58,GMT
ThisfunctionalityisprovidedbyVBGuestAdditions.TrytoreinstallGuestAdditions.Ifthat
doesn'thelp,youshouldlookforhelponVirtualBoxforums.
AnibalLopezsays:
August17,2014at10:32,GMT
HiSergie,
Havethingsworkingtilltheinstallofthedatabasesoftware.
Igottheerror:notenoughspace.
IhaveresizedthiswiththeVBoxManagerfromwithintheWindows8OS.
IhavealsohavetodoadiskresizingintheLinuxenvironmentwhenstartinguptheracvm's.
Doyoumayhaveanyadviceorhintonthis?
Thx,
Anibal
Davidsays:
August18,2014at05:01,GMT
Iamabeginner.Ifollowallyourstepsandfinishinstallwithnoproblem,thankyou.
Itrytoaddanode,butIcan'tfindaddNode.shin$ORACLE_HOME/oui/bin/.Couldyoutellme
howtorunaddNode.sh,appreciate.
SergeiRomanenko(http://www.lab128.com)says:
August18,2014at17:49,GMT
Onthefirstnode:
[oracle@rac1~]$.oraenv
ORACLE_SID=[grid]?+ASM1
[oracle@rac1~]$echo$ORACLE_HOME
/u01/app/11.2.0/grid

[oracle@rac1~]$ls$ORACLE_HOME/oui/bin/addNode.sh
/u01/app/11.2.0/grid/oui/bin/addNode.sh
says:
September19,2014at17:49,GMT
HiSergei,
Greatdocumentandnicedetailedsteps,thankyouverymuchforyoureffort:)
IwasabletodotheinstallIdidruninafewissues,butnothingthatpatienceandgooglesearch
couldnotsolve.
Thankyou,
Florin
says:
September28,2014at12:59,GMT
Hi,
Thanksforsharing.Thisisgold!Justwanttosharesomeexperiencewhilesettingup12.1.2
insteadof12.1.1.
1.Youneedalargerstorageforbothshareddiskandracvmsorelseyouwillrunintoissues
wheninstallingthedatabase.iuse10gbforshareddiskand30gbforracvms.Ibelievea30%
bumpwilldo(Ijustwanttobesafe)
2.Ihitanmemory_targetissuewhilestartinguptheRACvmafterthecreationofdatabase.I
resolveditbyincreasingmyvmmemoryto4gbandsettmpfsto3gb.Notsurewhatsthebest
waytoresolveitbutit'sworkingformeatm.
Thankyou,
Roger
says:
September28,2014at15:11,GMT
Roger,
Thankyouforthisvaluableinformation!
says:
September29,2014at20:37,GMT
SergeiRomanenko,
Whyyoudidn'tuseSCAN/DNSserver?
Thanks
Bhim
Sohansays:
October11,2014at10:10,GMT

Dowedon'tneedtodefineparameterin.bash_profileofbothnode?
syedaleemsays:
October24,2014at16:26,GMT
HiSergei:
Thankyouverymuchforsharingthisdocumentandgreatlyappreciatedforyoureffort.Iusedall
instructionsnicelylaidoutinthisdocumentandsetupmythetwonodeRACsuccessfully.
Thankyouagain.
SyedAleem.
Sohansays:
October29,2014at07:43,GMT
HiSyed,didyousetkernelparameterfortheconfigurationin/etcsysctl.conf?
RuiXianBAOsays:
December12,2014at23:49,GMT
RuiXianBAOsays:
December13,2014at00:02,GMT
HelloSergeiandothers,
Sorryfortheaboveemptypostamistake:)
BTWSergei,sincethefirsttimeIhadmoreerrorsthanexpectedafterinstallingtheGrid,thenI
deletedtheoldrac1andrac2andhavedoneareinstallation,butgotthesameerrorsasbelow.
Anyideawillbehighlyappreciated.
Best
RuiXian
INFO:INFO:ParsingKFOD00311:Errorscanningdevice/dev/sda
INFO:INFO:ParsingORA27041:unabletoopenfile
INFO:INFO:ParsingLinuxx86_64Error:13:Permissiondenied
INFO:INFO:ParsingAdditionalinformation:42
INFO:INFO:ParsingAdditionalinformation:140733744835552
INFO:INFO:ParsingKFOD00311:Errorscanningdevice/dev/sdb
INFO:INFO:ParsingORA27041:unabletoopenfile
INFO:INFO:ParsingLinuxx86_64Error:13:Permissiondenied
INFO:INFO:ParsingAdditionalinformation:42
INFO:INFO:ParsingAdditionalinformation:140733744835552
INFO:INFO:ParsingKFOD00311:Errorscanningdevice/dev/sda1
INFO:INFO:ParsingORA27041:unabletoopenfile
INFO:INFO:ParsingLinuxx86_64Error:13:Permissiondenied

INFO:INFO:ParsingAdditionalinformation:42
INFO:INFO:ParsingAdditionalinformation:140733744835552
INFO:INFO:ParsingKFOD00311:Errorscanningdevice/dev/sda2
INFO:INFO:ParsingORA27041:unabletoopenfile
INFO:INFO:ParsingLinuxx86_64Error:13:Permissiondenied
INFO:INFO:ParsingAdditionalinformation:42
INFO:INFO:ParsingAdditionalinformation:140733744835552
INFO:INFO:ParsingKFOD00311:Errorscanningdevice/dev/sdb1
INFO:INFO:ParsingORA27041:unabletoopenfile
INFO:INFO:ParsingLinuxx86_64Error:13:Permissiondenied
INFO:INFO:ParsingAdditionalinformation:42
INFO:INFO:ParsingAdditionalinformation:140733744835552
INFO:INFO:ParsingKFOD00316:ErroracquiringASMdiscoverystring:ORA29780:unableto
connecttoGPnPdaemon[CLSGPNP_ERR]
INFO:INFO:ParsingKFOD00105:Couldnotopenpfile'init@.ora'a
RuiXianBAOsays:
December19,2014at14:49,GMT
HelloSergeiandothers,
FYI,IgotthesameerrorevenwhenItriedudevinsteadofASMLib.Andmyfriendalsogetsthe
exactsameerror.Guessthisistheresultofthecurrentsoftwarecomponents'combination.In
theend,wedecidedgoingfarwardanywaydespiteoftheerrors.Andsurprisinglyallisthegood
intheend:)Thankyouagainforyourniceinstrucitons!
Best
RuiXian
Paul(www.webgeest.nl)says:
February27,2015at09:21,GMT
Hi,
Perfectdescription,stepbystepworkssmooth.
Except1thing:theSSHconnectivityduringInstallingDatabasestep7of13.
Beforedoinganythingonthisscreen,openupaterminalonbothnodes.
Checkthat:
*youcansshoracle@rac2ANDsshoracle@rac1twoways
*youcanalsosshtoyourself!sosshoracle@rac1fromrac1ANDsshoracle@rac2fromrac2.
Thismakestheentryforthe'own'machinecomeintotheknown_hosts
*makesurethattheauthorized_keyscontainsthepubkeysforBOTHhosts
Ifthisworksfine,thenONLYclicktheTestbutton.AndthenNext.DoNOTclickSetupbutton,
becauseIthinkthismessesuptheconfiguration.

Thankyouverymuchforthisexcellentpost!
Paul
rajsays:
March7,2015at15:40,GMT
Hi,
Thisisanawesomepost.Ididsetupsuccessfully.
Itwouldbeniceifyouwritepostsonnodeadditionandnodedeletionandstandbysetupin12c.
Thanks.
Rajansays:
April7,2015at13:15,GMT
WhiletakingbackupusingRMAN,gettingbelowerrorsdueto/dev/sdb1notmountedandneed
oracleuserpermissiontowriteit,
RMAN00569:===============ERRORMESSAGESTACKFOLLOWS===============
RMAN00571:===========================================================
RMAN03009:failureofbackupcommandonORA_DISK_1channelat04/07/201518:12:23
ORA19502:writeerroronfile
"/u01/app/oracle/product/12.1.0/dbhome_1/dbs/b_RACDB_0kq3q7jv_20_1",blocknumber
11648(blocksize=8192)
Plsadvise.Thanks.
EdgarRangelsays:
April13,2015at06:08,GMT
Sergei,
Thankyouverymuchforthisgreatguide!,IcreatedthelabsusingVirtualBoxinLinux(RedHat
EnterpriseLinuxServerrelease6.5),asperothercommentpostedhereIalsoused12.1.0.2,
buttogetitworkingIhadtoincreasememoryupto4gbpereachnode,andalsosettmpfsto
3gb.
Vaibhavsays:
June7,2015at13:10,GMT
Excellent
Dr.HussienSharaf(ite.frommasr.com)says:
August13,2015at19:28,GMT
Excellenttutorial.
IusedOracleLinux7for64bitandIfoundthatatthesectiontitled"UdevConfiguration"should
be
#/usr/lib/udev/scsi_idgud/dev/sdb
insteadof:
#/sbin/scsi_idgud/dev/sdb

Dr.HussienSharaf(ite.frommasr.com)says:
August16,2015at14:31,GMT
InOracleLinux7,disablingFirewall:
#systemctlstopfirewalld
#systemctldisablefirewalld
omerahmedsays:
October25,2015at20:30,GMT
Marcossays:
November5,2015at01:19,GMT
Ireallyappreciateyourposting.It'ssowelldetailedthatIhadalmostanyissue.Eventhe
expectederrormessageshelpedmetoconcludetheinstallation.Thanks!Spacibo!
TariqAhmadZafarsays:
December6,2015at15:56,GMT
TariqAhmadsays:
December6,2015at15:58,GMT
ExcellentArticle/GuidetoproceedRAC!
Oliviersays:
December15,2015at11:49,GMT
Onebasicquestion:Whyareyouusingbothasmlibandudev?Onlyoneofthemshouldbe
needed.
abisays:
January18,2016at03:55,GMT
SuperArticle.ThanksaTon.Reallyuseful.IwasabletosetupracdbproperlyforwhichIwas
strugglingformanydays.
Kirksays:
February15,2016at04:52,GMT
Whenirunthecommandfdisk/dev/sdbienterthecorrectitemsbutwhenigetthefollowing:
firstsector20488388607,default2048
lastsector20488388607,default8388607
igowiththedefaults.Thenwhenirunthefollowingcommand
oracleasmcreatediskDISK1/dev/sdb1
igetfailedoninstantiatingdisk
Whatdididowrong?
Wusays:
March14,2016at19:32,GMT
Somehowtheinstructionsinthelastpartofsection"CreateSharedDisks"didnotworkforme.I
couldnotseeanythingwhenIused"Chooseexistingdisk"optiontoattachthenewlycreated
shareablevirtualdisktotheotherVM.canIgetsomehelp?thanks!

Leaveyourcomment:

YourName*
Email
Website
(includehttp://...)
Thisisasimpleantispammeasure.Pleasesolvethemathproblem:(1211)*1485=

PostComment

Copyright128ConsultingLLC