Crac|e c|uster on CentCS us|ng CentCS c|usterware

Introduction
This document will cover the setup/conIiguration oI CentOS cluster and Oracle 11g database as a cluster
resource.

Download Software and assumptions
1. CentOS 5.6 64bit
2. Oracle Database 11g R1 binaries
3. You should have internet access Ior yum install
4. ILO cards should have been conIigured Ior Iencing
5. Both CentOS nodes should communicate with each
other with: 1 public interIace (bonding is suggested)
1 private interconnect (bonding is suggested)
SAN storage LUNs should be accessible to both nodes, as cluster storage.

Modify /etc/hosts in both nodes
/etc/hosts Iile in each server should be modiIied to point to ilos, private interIaces and public interIaces.
See the example bellow.
[rootdb2 ~j# more /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1 localhost.localdomain localhost
#..1 localhost6.localdomain6 localhost6 ## comment this line
192.168.0.70 db1.dyndns.org ## 1his is the public hostname for db1
192.168.2.70 db1 ## 1his is the private hostname for db1
192.168.0.80 db2.dyndns.org ## 1his is the public hostname for db2
192.168.2.50 db2 ## 1his is the private hostname for db2
# Cluster JIP for Oracle
192.168.0.90 oraprd.dyndns.org ##Cluster Jirtual Host
192.168.0.151 ilodb1 ## Ilo1
192.168.0.152 ilodb2 ##Ilo 2
[rootdb2 ~j#

Discovering the needed storage
PerIorm: Idisk l to check the attached disks. Your storage admin should have assigned those disks to
both nodes. A reboot oI the servers maybe necessary to see the new disks.
lsscsi rpm installation in both nodes, must be perIormed.

Check lsscsi output in both nodes. Should display the same disks.

BeIore the Reboot, you may try the Iollowing command, to discover the disks LUNs online (should run in
both nodes)
echo "- - -" ~ /sys/class/scsihost/host3/scan
Where host3 can be replaced by the output oI the command:
lsscsi H (this command will display all fiber cards connected to your server)
[rootdb11 ~]# lsscsi -H
[0] NULL~ This means that the fiber card has the instance 0, so it would be host0
You must have already opened the /var/log/messages in each node, Irom another putty session, because
you will probably see the attached new devices there, the moment they are discovered by the previous
command.

reate the disk layout (LVM) - This is to be done only in one node
Create 4 pvs. This conIiguration is enough Ior a simple Oracle database installation, with no heavy usage.
II a device, has previous partitions because oI previous usage (e.g Solaris), then beIore the pv creation,
you need to delete those partitions. Use Idisk command to connect to the device.










reate the volume Groups
Create vgorabin, vgoradata, vgredo1, vgredo2













reate the Logical Volumes
Create lvorabin, lvoradata, lvredo1, lvredo2














reate the file systems - If a logical volume is inactive, you cannot create the filesystem, so then you
must try to activate with: lvchange -ay lv_name





















Reboot the other node, to read all LVM changes (this is needed, according to my experience)
[rootdb2 ~]# shutdown -ry now
AIter the reboot:













Install all cluster and cluster storage rpms (IN BOTH NODES)
This includes cman, rgmanager, clvmd, ricci, luci, system-conIig-cluster and lvm2-cluster








Start the cluster config gui (you may use luci web interface to build the cluster, you have to start
ricci and luci services first)















Click on Create New ConIiguration

Name the cluster, do not use quorum yet, click ok



Click on Add a Cluster Node, put the private hostname (the one used Ior the interconnect), choose 1 vote,
click Ok

Click again on Add a Cluster Node, put the other node`s hostname (interconnect), choose 1 vote, click Ok

Click on Failover Domains, Create a Failover Domain

Name the Domain as oracledomain and click Ok



Click on Available Cluster Nodes and choose both nodes



Click on both check boxes

And adjust priority (priority is the preIerable node Ior the cluster service) to db11, db22, then click
Close


Click on Resources and then on Create Resource (4 times Ior Iile systems, 1 Ior VIP). Use the logical
volumes created beIore. Usually we use also similar resource names Ior the mount points, as the name oI
the Iilesystem.
/orabin ORACLEHOME
/oradata DataIilesControlFilesTemporary
/redo1, /redo2 RedoLogs (The setup oI oracle redologs multiplex as it is called- has to be done later by
the Oracle DBA and can be done online. For our installation all redolog Iiles will be saved in /oradata/..)
/arch iI database is to be conIigured in archive log mode, Ior online backups (not our case, Oracle DBA
job also). II you know Irom the beginning oI the project, that the database, will not be in archive log
mode, then you do not need this Iile system at all)












Now Create the VIP IP resource and click Ok. The VIP should exist in /etc/hosts on both nodes. Check
the monitor Link checkbox.



Click on Services and then on Create a Service


Choose TESTDB as servicename and click Ok


Make sure the utostart This Service and Restart options are checked. Click on Failover Domain and
choose oracledomain














Click on Add a shared resource Ior this service (5 times, one Ior each resource). You must choose Iirst the
VIP and then the Iile systems


Now click on the VIP resource and then on the Attach a Shared Resource to the selection (4 times, one Ior
each Iile system)








Click Close button




Go to FileSave

Press Ok on the deIaults



Close the system-conIig-cluster GUI













Enable automatic startup oI cluster daemons (in both nodes db1 and db2)


Reboot both nodes and check (Irom the ILO console) that cluster is Iunctional.
When both systems are up, perIorm the command clustat to veriIy the cluster proper conIiguration.













Create the quorum conIiguration (with direct editing oI the cluster.conI), on none db1. Then propagate to
db2 with ccstool
Create a quorum device with mkqdisk. Chose an empty device (normally quorum devices should be less
than 1 GB oI capacity).



VeriIy the quorum on node2 also

ModiIy the cluster.conI on db1 with the three white shadow lines (add them under the second line)

ModiIy the twonode1 and change it to twonode0 (twonode1 can be used when we have no
quorum)

Change the version in cluster.conI

Save the cluster.conI and propagate to db2















ConIigure the ILO cards on both nodes. Connect to ILO with the Web interIace and make sure that you
have setup a Ience user/password. The user should have the right to connect and reset the server. ILO
cards should be connected to a private switch, accessible by the sys admin Ior maintenance. The
Iollowing printscreens, shows ILO conIig on one node. The same conIig should be done on the other ILO
also.


Update cluster.conI on db1 with the ILO Ience inIormation. Insert the lines in white shadow. In our case,
we used ilodb1 and ilodb2 Ior ILO hostnames, and we have updated /etc/hosts in both nodes with the
ILO IPs.

ModiIy cluster.conI on db1 and add the lines in white shadow.



Change version


Save and propagate to db2

Test connectivity with ilodb1, ilodb2, db1, db2, VIP, GATEWAY Irom both nodes
[rootdb1 ~j# ping ilodb1
PING ilodb1 (192.168.0.151) 56(84) bytes of data.
64 bytes from ilodb1 (192.168.0.151). icmpseq1 ttl64 time1.83 ms
64 bytes from ilodb1 (192.168.0.151). icmpseq2 ttl64 time0.580 ms
--- ilodb1 ping statistics ---
2 packets transmitted, 2 received, 0 packet loss, time 1000ms
rtt min/avg/max/mdev 0.580/1.207/1.834/0.627 ms
[rootdb1 ~j# ping ilodb2
PING ilodb2 (192.168.0.152) 56(84) bytes of data.
64 bytes from ilodb2 (192.168.0.152). icmpseq1 ttl64 time1.50 ms
64 bytes from ilodb2 (192.168.0.152). icmpseq2 ttl64 time0.511 ms
--- ilodb2 ping statistics ---
2 packets transmitted, 2 received, 0 packet loss, time 1000ms
rtt min/avg/max/mdev 0.511/1.006/1.501/0.495 ms
[rootdb1 ~j# ping 192.18..1
PING 192.168.0.1 (192.168.0.1) 56(84) bytes of data.
64 bytes from 192.168.0.1. icmpseq1 ttl64 time0.233 ms
--- 192.168.0.1 ping statistics ---
1 packets transmitted, 1 received, 0 packet loss, time 0ms
rtt min/avg/max/mdev 0.233/0.233/0.233/0.000 ms
[rootdb1 ~j# ping db1
PING db1 (192.168.2.70) 56(84) bytes of data.
64 bytes from db1 (192.168.2.70). icmpseq1 ttl64 time0.063 ms
--- db1 ping statistics ---
1 packets transmitted, 1 received, 0 packet loss, time 0ms
rtt min/avg/max/mdev 0.063/0.063/0.063/0.000 ms
[rootdb1 ~j# ping db2
PING db2 (192.168.2.50) 56(84) bytes of data.
64 bytes from db2 (192.168.2.50). icmpseq1 ttl64 time0.178 ms
--- db2 ping statistics ---
1 packets transmitted, 1 received, 0 packet loss, time 0ms
rtt min/avg/max/mdev 0.178/0.178/0.178/0.000 ms
[rootdb1 ~]#
[rootdb1 ~j# ping 192.18..9
PING 192.168.0.90 (192.168.0.90) 56(84) bytes of data.
64 bytes from 192.168.0.90. icmpseq1 ttl64 time0.065 ms
64 bytes from 192.168.0.90. icmpseq2 ttl64 time0.029 ms
--- 192.168.0.90 ping statistics ---
2 packets transmitted, 2 received, 0 packet loss, time 1000ms
rtt min/avg/max/mdev 0.029/0.047/0.065/0.018 ms

Reboot both nodes, for the quorum device to start properly and to count in voting. qdiskd daemon
should always start before the cman.
[rootdb1 ~]# shutdown -ry now (reboot iI it is going too slowly)
[rootdb2 ~]# shutdown -ry now (reboot iI it is going too slowly)
Note: Db1 and db2 are going to fence each other once, until they join to the cluster. 1ust wait. In
some cases you may have to update /etc/cluster/cluster.conf with
<fence_daemon post_fail_delay"0" post_join_delay"120"/>
Check cluster status, aIter both systems are up and running.









Oracle installation steps
Create user groups oinstall and dba, then create user oracle. Make sure they share the same userId and
groupIDs between both nodes. This has to be done in both cluster nodes.

ModiIy .bashproIile in /home/oracle and add the Iollowing inIo (in both nodes). /home/oracle is
assumed to be the home directory oI user oracle.
export ORACLEHOME/orabin/oracle/product/11.2.0/TESTDB
export ORACLEBASE/orabin/oracle
export ORACLESIDTESTDB
export PATH$ORACLEHOME/bin.$PATH

ModiIy /etc/sysctl.conI in both nodes, and add (at the end oI the Iile):
fs.file-max 6815744
net.ipv4.iplocalportrange 9000 65500
net.core.rmemdefault 262144
net.core.wmemdefault 262144
net.core.wmemmax 1048576
fs.aio-max-nr 1048576
kernel.sem 250 32000 100 128
Perform sysctl -p for the kernel to re-read the file online (both nodes)

ModiIy /etc/security/limits.conI and add (both nodes, end oI the Iile):
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536


1E: The minimum swap space you need is 2GBs in both servers.



Change recursively the ownerships Ior all cluster Iile systems to belong to oracle.oinstall (both nodes)
[rootdb2 /]# chown -R oracle.oinstall /orabin/ /oradata/ /redo1 /redo2
[rootdb1 /]# chown -R oracle.oinstall /orabin/ /oradata/ /redo1 /redo2
ModiIy the /etc/proIile and add the Iollowing (both nodes, end oI Iile) lines (shown with white shadow)





Add the Iollowing line to the /etc/pam.d/login Iile, iI it does not already exist.
session required pam_limits.so







Disable SELINUX on both nodes. Change the value to: disabled Ior the parameter SELINUX in the Iile
/etc/selinux/conIig. This is Ior the chamge to be there aIter a system reboot. To change the SELINUX
while the systems are online, do the Iollowing (both nodes)

echo > /selinux/enforce


Move the oracle binary cds to /home/oracle and unzip them both (as user oracle) on node db1

Install the Iollowing rpms (or greater versions), Ior Oracle to install properly (on both nodes)
binutils-2.17.50.0.6
compat-libstdc-33-3.2.3
compat-libstdc-33-3.2.3 (32 bit)
elIutils-libelI-0.125
elIutils-libelI-devel-0.125
gcc-4.1.2
gcc-c-4.1.2
glibc-2.5-24
glibc-2.5-24 (32 bit)
glibc-common-2.5
glibc-devel-2.5
glibc-devel-2.5 (32 bit)
glibc-headers-2.5
ksh-20060214
libaio-0.3.106
libaio-0.3.106 (32 bit)
libaio-devel-0.3.106
libaio-devel-0.3.106 (32 bit)
libgcc-4.1.2
libgcc-4.1.2 (32 bit)
libstdc-4.1.2
libstdc-4.1.2 (32 bit)
libstdc-devel 4.1.2
make-3.81
sysstat-7.0.2
unixODBC-2.2.11
unixODBC-2.2.11 (32 bit)
unixODBC-devel-2.2.11
unixODBC-devel-2.2.11 (32 bit)
You may check with the Iollowing command, to veriIy that the rpm is already installed (greater versions
are ok). The Iollowing are some examples Ior a Iew rpms.
[rootdb1 oraclej# rpm -qa[grep -i sysstat
sysstat-7.0.2-11.el5
[rootdb1 oraclej# rpm -qa[grep -i libaio
libaio-devel-0.3.106-5
libaio-.3.1-5 1HIS SHS 1HA1 B1H 32bit A 4bit ARE IS1ALLE BECAUSE I1
IS ISPLAYE 1ICE
libaio-.3.1-5 1HIS SHS 1HA1 B1H 32bit A 4bit ARE IS1ALLE BECAUSE I1
IS ISPLAYE 1ICE
libaio-devel-0.3.106-5
[rootdb1 oraclej# rpm -qa[grep -i libstdc++
compat-libstdc-33-3.2.3-61
libstdc-devel-4.1.2-51.el5
compat-libstdc-33-3.2.3-61
libstdc-4.1.2-51.el5
libstdc-4.1.2-51.el5



II it is not installed then perIorm yum install rpm-name Ior all the above rpms.


NOTE: Another way to Iind and install the necessary rpms (if for example Internet is not accessible),
would be the CentOS DVD. You have to insert the DVD/CD into the CDROM, then go to /media/cdrom
(check where the CD will auto mount itselI), and then move to the directory Server. In this directory you
should Iind the rpms that you should install in both systems.

Use the command: rpm -ivh rpm-name to install
rpm -qa [ grep -i rpm-name to verify the existence




To install Oracle database in the cluster Virtual Hostname (oraprd.dyndns.org), we do the Iollowing trick.
a. Login as root in db1 and issue: hostname oraprd.dyndns.org
b. Login as oracle user, and start the installer, on node db1 (cd /home/oracle/database):
c. ./runInstaller (Don`t Iorget to have Xwin running on your laptop, and to export
DISPLAYyourIP:0.0)
d. AIter the installation we reverse back the hostname to db (as user root)
e. This trick is made only Ior oracle dbconsole to work properly in both nodes. Dbconsole binds
itselI with the actual hostname, and cannot be changed later.





Say yes to the warning regarding the email address













Click on Fix & Check again. Then go to /tmp/CVU11.2.0.1.0oracle and run (as root):
./runfixup.sh


Click Ok to continue




Click on Iinish. DO NOT FORGET TO OPY THE FIXES TO THE OTHER NODE LSO
(db2). In our case, the problem was only two values in /etc/sysctl.conI We should copy these
values to db2, in the Iile /etc/sysctl.conI and perIrom sysctl p. The values are: kernel.sem 250
100.
Probably, iI you use the values mentioned earlier (page 56), Ior the Iile sysctl.conI, you will have
no problem.


















II the oracle checks has no anymore Iindings, you can click on Finish








You may view the install log with the Iollowing command, all the time:




When prompted, run the Iollowing scripts as root, and click Ok








Make sure that all entries in /etc/sysctl.conI Iile Irom db1 are identical oI those in db2. II necessary, copy
the Iile Irom db1 to db2 and do: sysctl -p

Last lines from file /etc/sysctl.conf, in node db1 - opy them to db2 also and do : sysctl -p
# Controls the maximum number of shared memory segments, in pages
kernel.shmall 4294967296
fs.file-max 6815744
net.ipv4.iplocalportrange 9000 65500
net.core.rmemdefault 262144
net.core.wmemdefault 262144
net.core.wmemmax 1048576
fs.aio-max-nr 1048576
kernel.sem 250 32000 100 128

Login as oracle and run netca (Network ConIiguration Assistant)




































To create the Oracle Database, login as oracle and run dbca (Database ConIiguration Assistant)





Use the same passwords Ior all Oracle sys accounts: abc123






lick on Sample Schemas only if it is a test database

Accept the deIault memory distribution This is normally good Ior most installations (II you need more
memory Ior other resources, you may decrease the percentage to 30

Oracle processes are the maximum number oI connections allowed to connect to the Oracle server. This is
DBA decision again, and depend on the project.



Click on Finish


Click on Ok





Click on Exit


Copy /etc/oratab and /etc/oraInst.loc Iiles, Irom db1 to db2


Switch hostname back to normal (Irom oraprd to db1)
rootoraprd etc#hostname db1 (as user root)
Create a TESTDB.sh script on /etc/init.d on both nodes. This is the script used Irom the cluster to
stop/start/status the TESTDB service. You can view this script Irom the /etc/init.d/TESTDB.sh












Open system-conIig-cluster and ad the TESTDB.sh script resource



Click on Edit Service Properties, choose the last resource (redo2) and click on Attached a Share Resource
to the selection.

Select the TESTDB.sh script and click Ok




Click on Send to Cluster




Create the Iollowing soIt links (as oracle user), in order Ior the oracle dbconsole to properly work. We
use the real hostnames (db1, db2) and we link them to the Virtual host name (oraprd).
cd /orabin/oracle/product/11.2.0/TESTDB/oc4j/j2ee
ln -s OC4JDBConsoleoraprd.dyndns.orgTESTDB/ OC4JDBConsoledb1TESTDB
ln -s OC4JDBConsoleoraprd.dyndns.orgTESTDB/ OC4JDBConsoledb2TESTDB

cd /orabin/oracle/product/11.2.0/TESTDB
ln -s oraprd.dyndns.orgTESTDB/ db1TESTDB
ln -s oraprd.dyndns.orgTESTDB/ db2TESTD

As user oracle, go to $ORACLEHOME/network/admin and insert the lines in White shadow, into the
Iile listener.ora. Save the Iile and then do: lsnrctl reload




Also, in the same directory as beIore, vi the Iile tnsnames.ora and change the SERVIE_NME to
show SID


PerIorm two Iailover tests, as user root on the TESTDB service, to check Cluster Iunctionality
clusvsadm -r TESTDB
clustat
cman_tool status
cman_tool nodes









Login into the database (via local IireIox Irom db1 or db2), and check database health (DBA task).
Passwords Ior sys/system accounts is abc123







END OF INSTLLTION