Installing Oracle 12.1.0.

2 RAC on Oracle Linux 7-part 1

Fuente:

https://martincarstenbach.wordpress.com/2015/01/05/installing-oracle-12-1-0-2-rac-on-oracle-linux-7-part-1/

Now that 12.1.0.2 is certified on RedHat Linux 7 and spin-off environments its time to test the
installation of RAC on such a system.

The installation of the OS is different from Oracle Linux 5 and 6-with these distributions was
very straight forward how to install the operating system the method has changed significantly in
release 7. I wont cover the complete installation here, as always Tim Hall was quicker than me,
but it makes me wonder who signed off the user interface for the partitioning wizard I
personally think that the kickstart partitioning-information is a lot easier to understand.

http://oracle-base.com/articles/linux/oracle-linux-7-installation.php

After walking through the steps in the graphical installer I thought that Id rather kickstart the
whole lot. If you have not yet used kickstart to create Oracle/Red Hat/CentOS hosts then now is
the time to start admiring the technology!

My host is Oracle Linux 6.6 and there arent any packages that arent part of the distribution,
with one exception: I have compiled dnsmasq from the current source since it had a bug when
specifying the bind-interfaces command line option if other dnsmasq processes were already
running. libvirt uses dnsmasq extensively for its own networking (I since then upgraded the host
to Oracle Linux 7 too and the problem is no longer present)

I dont like repetitive tasks

Anything that is repetitive is not the best use of anyones time, especially not mine. I prefer to
run a single command, do something different and come back with the OS already installed. For
the installation to work I am relying on PXE booting to pass the kernel and initial RAM disk.
TFTP is in charge of that. A Kickstart file is used for the silent installation. Lets start off with the
Pre Execution Boot Environment (I am putting a shameless plug in here for my 12c book which
covers this subject extensively if you need more background). Here is my default configuration
file in pxelinux.cfg:

default menu.c32
prompt 0

menu title PXE Boot Menu

label oracle-linux 6.6 x86-64

menu label oracle-linux 6.6 x86-64
kernel /tftpboot/webroot/images/ol/6/6/vmlinuz
append initrd=/tftpboot/webroot/images/ol/6/6/initrd.img
ks=http://192.168.150.1/ol66.ks ksdevice=eth0
label oracle-linux 7.0 x86-64
menu label oracle-linux 7.0 x86-64
kernel /tftpboot/webroot/images/ol/7/0/vmlinuz
append initrd=/tftpboot/webroot/images/ol/7/0/initrd.img
inst.ks=http://192.168.150.1/ol70.ks inst.repo=http://192.168.150.1/ol70

The next step is to ensure that you have the images you are pointing to.
[root@ol62 0]# cp -iv /media/ol70/images/pxeboot/* .
`/media/ol70/images/pxeboot/initrd.img' -> `./initrd.img'
`/media/ol70/images/pxeboot/TRANS.TBL' -> `./TRANS.TBL'
`/media/ol70/images/pxeboot/upgrade.img' -> `./upgrade.img'
`/media/ol70/images/pxeboot/vmlinuz' -> `./vmlinuz'
[root@ol62 0]# pwd
/tftpboot/webroot/images/ol/7/0

The Pre Execution Environment (PXE) will supply the KS (kickstart) directive to the boot
process. The Initial RAMdisk and kernel will be pulled via tftp-dont forget to enable xinetd and
tftp if you are not using dnsmasq like me or this wont work. You also need a DHCP server that
provides the NIC of the VM with an IP and the reference to the PXE boot configuration. If you
are using a lab server then dnsmasq can do this for you. I believe that admins in real life might
not do it that way The official Red Hat documentation has a nice section on how to configure
your environment for pxeboot with the ISC DHCP package, too. Here is the pxe.conf file I used
successfully (with that newly compiled dnsnasq-again the stock Oracle 6.x version cant handle
the bind-interface at the time of writing.
[root@ol62 ~]# cat /etc/dnsmasq.d/pxe.conf
tftp-root=/tftpboot/webroot
enable-tftp

dhcp-boot=pxelinux.0
dhcp-option=vendor:PXEClient,6,2b
dhcp-no-override

pxe-prompt="Press F8 for boot menu", 3

pxe-service=X86PC, "Boot from network", pxelinux
pxe-service=X86PC, "Boot from local hard disk", 0

#log-queries
#log-dhcp

domain=example.com
dhcp-range=192.168.150.50,192.168.150.150,12h

This file then went into /etc/dnsmasq.d/ as you can see, and I restarted the daemon to make it
read the file. Initially I found the log-queries and log-dhcp output quite useful to work out what
was happening, you can see these messages in /var/log/messages. There are many so you
might want to turn that off as soon as you fixed any potential problems.

Kickstarting

My kickstart file assumes that the installation tree is available via http from 192.168.150.1/ol70.
Its just a loopback mounted ISO image so easy to implement in my host.
My kickstart file is not really special, it simply sets the partitioning schema creating 2 volume
groups: rootvg for system related things such as swap, boot and root plus oraclevg for the oracle
binaries. I based it primarily on the anaconda-ks.cfg file in /root that has been created during the
installation. The kickstart documentationis also rather nice. Since there is no preinstall RPM
available right now I resorted to a %post script to add the Oracle-specific settings:

UPDATE

Since I wrote this post Oracle Linux 7.1 has been released which features the 11.2 and 12.1
RDBMS preinstall RPMs rendering some of the steps shown here obsolete.

%post --log=/root/ks-post.log

cat >> /etc/sysctl.conf <<EOF

fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500
kernel.panic_on_oops = 1
EOF

sysctl -f --system

cat >> /etc/security/limits.conf <<EOF

oracle soft nofile 1024
oracle hard nofile 65536
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft stack 10240
oracle hard stack 32768
oracle soft memlock 60397977
oracle hard memlock 60397977

grid soft nofile 1024

grid hard nofile 65536
grid soft nproc 2047
grid hard nproc 16384
grid soft stack 10240
grid hard stack 32768
grid soft memlock 60397977
grid hard memlock 60397977
EOF

cat > /etc/udev/rules.d/61-asm.rules <<EOF

KERNEL=="vd[cdefgh]1", OWNER="grid", GROUP="asmdba" MODE="0660"
EOF
gid=10000
for group in asmdba asmadmin kmdba dgdba backupdba dba oinstall; do
groupadd -g $(( gid += 1 )) $group ;
done

useradd -u 10000 -g oinstall -G asmdba,kmdba,dgdba,backupdba,dba oracle

useradd -u 10001 -g oinstall -G asmdba,asmadmin,dba grid

mkdir -p /u01/app/oracle
mkdir -p /u01/app/grid

chown -R grid:oinstall /u01

chown -R oracle:oinstall /u01/app/oracle

This is of course just an example-you need to adjust this to your environment. I am assuming a
separation of duties here as well which not everyone will (and should!) apply in his environment.
There are also quite a few individual packages in the %package list right now that Ill replace
with the call to the preinstall RPM once its available. Oracle Linux 7 reflects the fact that many
system daemons use a conf.d directory to separate customisations from the main configuration
file. Oracle 12.1.0.2 does not check for this: when I tried moving my Oracle specific code into
/etc/sysctl.d/50-oracle.conf OUI could not find and parse the file. It subsequently complained
about an unknown configuration. So therefore we are back to the old days.

Now all I need to do is to call virt-install on the command line and wait for the KVM
environment to be created.

References

oracle-base.com enough said. Had the OS install first, and the installation of the RDBMS.
Kickstart reference for Red Hat Linux 7

Boot options can be found here they have changed a bit in release 7!

I have documented the whole process in more detail in Expert Consolidation in Oracle 12c

Installing Oracle 12.1.0.2 RAC on Oracle Linux 7-part 2

https://martincarstenbach.wordpress.com/2015/01/12/installing-oracle-12-1-0-2-rac-on-oracle-linux-7-part-2/

Posted by Martin Bach on January 12, 2015

In the first part of the article series you could read how a kickstart file made the installation of
Oracle Linux 7 a lot more bearable. In this part of the series its all about configuring the
operating system. The installation of Grid Infrastructure and the Oracle database is for another
set of posts.
There are quite some differences between Oracle Linux 6 and 7

To me the transition from Oracle Linux 6 to 7 feels like the step from Solaris 9 to 10 at the time.
Personally I think that a lot has changed. Although, its fair to say that it has been quite some
time it has been announced that the network stack commands we know and love are deprecated
and might go Even with Oracle Linux 6 there was a threat that network manager would now
be the only tool to modify your network settings (which thankfully was not the case). A lot of
efforts of the Linux community have now come to fruition, and its time to adjust to the future.
Even when its painful (and it is, at least a bit).

Configuring the network

The warning has been out there quite a while but now it seems to be true-no more system-config-
network-tui to configure the network! No more ifconfig! Oh dear-quite a bit of learning to be
done. Luckily someone else has done all the legwork and documented the changes. A good
example is this one:

https://dougvitale.wordpress.com/2011/12/21/deprecated-linux-networking-commands-and-their-
replacements/

So first of all-dont fear: although all network interfaces are configured using network manager
now, you can still use a command line tool: nmtui. After trying it out I have to say Im not really
convinced about its usability. What appears better is the use of the nmcli, network manager
command line tool. Its use is quite confusing, and it appears to me as if the whole network
manager toolset was developed for laptop users, not servers. But I digress. I have a few
interfaces in my RAC VM, the first was configured during the installation, eth[1-3] arent
configured yet.

[root@localhost ~]# nmcli connection show

NAME UUID TYPE DEVICE
System eth0 77e3f8a9-76d0-4051-a8f2-cbbe39dab089 802-3-ethernet eth0
[root@localhost ~]# nmcli device status
DEVICE TYPE STATE CONNECTION
eth0 ethernet connected System eth0
eth1 ethernet disconnected --
eth2 ethernet disconnected --
eth3 ethernet disconnected --
lo loopback unmanaged --
[root@localhost ~]#

At this point I have used eth0 as the management network (similar to the way Exadata does) and
will use the other networks for the database. eth1 will act as the public network, eth2 and eth3
will be private.

Although the network interfaces can be named differently for device name persistence I stick
with the old naming for now. I dont want to run into trouble with the installer just yet. On
physical hardware you are very likely to see very different network interface names, the kernel
uses a naming scheme identifying where the cards are (on the main board, or in extension cards
for example). Ill write another post about that soon.

Using dnsmasq (on the host) I configure my hosts for these addresses:

[root@ol62 ~]# grep rac12pri /etc/hosts

192.168.100.107 rac12pri1.example.com rac12pri1
192.168.100.108 rac12pri1-vip.example.com rac12pri1-vip
192.168.100.109 rac12pri2.example.com rac12pri2
192.168.100.110 rac12pri2-vip.example.com rac12pri2-vip
192.168.100.111 rac12pri-scan.example.com rac12pri-scan
192.168.100.112 rac12pri-scan.example.com rac12pri-scan
192.168.100.113 rac12pri-scan.example.com rac12pri-scan

Configuring the interface is actually not too hard once you got the hang of it. It took me a little
while to get it though It almost appears as if something that was simple and easy to use was
made difficult to use.
[root@localhost ~]# nmcli con add con-name eth1 ifname eth1 type ethernet ip4
192.168.100.107/24 gw4 192.168.100.1
[root@localhost ~]# nmcli con add con-name eth2 ifname eth2 type ethernet ip4
192.168.101.107/24
[root@localhost ~]# nmcli con add con-name eth3 ifname eth3 type ethernet ip4
192.168.102.107/24

[root@localhost ~]# nmcli con show

NAME UUID TYPE DEVICE
eth2 ccc7f592-b563-4b9d-a36b-2b45809e4643 802-3-ethernet eth2
eth1 ae897dee-42ff-4ccd-843b-7c97ba0d5315 802-3-ethernet eth1
System eth0 77e3f8a9-76d0-4051-a8f2-cbbe39dab089 802-3-ethernet eth0
eth3 b6074c9a-dcc4-4487-9a8a-052e4c60bbca 802-3-ethernet eth3

I can now verify the IP addresses using the ip tool (ifconfig was not installed, I havent yet
checked if there was a compatibility package though)

[root@localhost ~]# ip addr show

1: lo: &lt;LOOPBACK,UP,LOWER_UP&gt; mtu 65536 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast
state UP qlen 1000
link/ether 52:54:00:6e:6f:67 brd ff:ff:ff:ff:ff:ff
inet 192.168.150.111/24 brd 192.168.150.255 scope global eth0
inet6 fe80::5054:ff:fe6e:6f67/64 scope link
valid_lft forever preferred_lft forever
3: eth1: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast
state UP qlen 1000
link/ether 52:54:00:96:ad:88 brd ff:ff:ff:ff:ff:ff
inet 192.168.100.107/24 brd 192.168.100.255 scope global eth1
inet6 fe80::5054:ff:fe96:ad88/64 scope link
 valid_lft forever preferred_lft forever
4: eth2: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast
state UP qlen 1000
link/ether 52:54:00:c1:cc:8e brd ff:ff:ff:ff:ff:ff
inet 192.168.101.107/24 brd 192.168.101.255 scope global eth2
inet6 fe80::5054:ff:fec1:cc8e/64 scope link
valid_lft forever preferred_lft forever
5: eth3: &lt;BROADCAST,MULTICAST,UP,LOWER_UP&gt; mtu 1500 qdisc pfifo_fast
state UP qlen 1000
link/ether 52:54:00:7e:59:45 brd ff:ff:ff:ff:ff:ff
inet 192.168.102.107/24 brd 192.168.102.255 scope global eth3
inet6 fe80::5054:ff:fe7e:5945/64 scope link
valid_lft forever preferred_lft forever

Now whats left is setting the hostname-which is a simple call to hostnamectl static set-
hostname rac12pri1. nmcli gives you an interface to changing the hostname as well. I repeated
the steps for node 2, they are identical except for the network IP addresses of course.

So that concludes the network setup.

Managing linux daemons

If you are curious about setting services at runlevel, then therell be another surprise:

[root@rac12pri2 ~]# chkconfig --list

Note: This output shows SysV services only and does not include native
systemd services. SysV configuration data might be overridden by native
systemd configuration.

If you want to list systemd services use 'systemctl list-unit-files'.

To see services enabled on particular target use
'systemctl list-dependencies [target]'.

iprdump 0:off 1:off 2:on 3:on 4:on 5:on 6:off

iprinit 0:off 1:off 2:on 3:on 4:on 5:on 6:off
iprupdate 0:off 1:off 2:on 3:on 4:on 5:on 6:off
netconsole 0:off 1:off 2:off 3:off 4:off 5:off 6:off
network 0:off 1:off 2:on 3:on 4:on 5:on 6:off
pmcd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
pmie 0:off 1:off 2:off 3:off 4:off 5:off 6:off
pmlogger 0:off 1:off 2:off 3:off 4:off 5:off 6:off
pmmgr 0:off 1:off 2:off 3:off 4:off 5:off 6:off
pmproxy 0:off 1:off 2:off 3:off 4:off 5:off 6:off
pmwebd 0:off 1:off 2:off 3:off 4:off 5:off 6:off
rhnsd 0:off 1:off 2:on 3:on 4:on 5:on 6:off
[root@rac12pri2 ~]#

If you just got familiar with upstart then there are some bad news: upstart is now replaced with
systemd This might be the right time to read up on that if you arent familiar with it yet:
https://access.redhat.com/documentation/en-
US/Red_Hat_Enterprise_Linux/7/html/System_Administrators_Guide/chap-
Managing_Services_with_systemd.html

Things are a little different with that, so here is an example how to enable and start the NTP
service. It has to be installed first if that hasnt been the case. You also should add the -x flag
in /etc/sysconfig/ntpd. First I would like to see if the service is available. You use systemctl for
this-so instead of a chkconfig ntpd list you call systemctl as shown:

[root@rac12pri ~]# systemctl list-units --type service --all | grep ntpd

ntpd.service
loaded inactive dead Network Time Service
ntpdate.service
loaded inactive dead Set time via NTP

I have to get used to the new syntax: previously you used service <whatever> status and then,
if you needed, typed backspace a few times and changed status to start. The new syntax is closer
to human language but less practical: systemctl status <service>. Changing status to start
requires more typing.

The check proved that the service exists (i.e. the NTP package is installed), but it is not started.
We can change this:

[root@rac12pri ~]# systemctl enable ntpd.service

[root@rac12pri ~]# systemctl start ntpd.service
[root@rac12pri ~]# systemctl status ntpd.service
ntpd.service - Network Time Service
Loaded: loaded (/usr/lib/systemd/system/ntpd.service; enabled)
Active: active (running) since Tue 2014-12-16 15:38:47 GMT; 1s ago
Process: 5179 ExecStart=/usr/sbin/ntpd -u ntp:ntp $OPTIONS (code=exited,
status=0/SUCCESS)
Main PID: 5180 (ntpd)
CGroup: /system.slice/ntpd.service
5180 /usr/sbin/ntpd -u ntp:ntp -g -x

Dec 16 15:38:47 rac12pri ntpd[5180]: Listen normally on 8 eth1

fe80::5054:ff:fe96:ad88 UDP 123
Dec 16 15:38:47 rac12pri ntpd[5180]: Listen normally on 9 eth2
fe80::5054:ff:fec1:cc8e UDP 123
Dec 16 15:38:47 rac12pri ntpd[5180]: Listen normally on 10 eth3
fe80::5054:ff:fe7e:5945 UDP 123
Dec 16 15:38:47 rac12pri ntpd[5180]: Listen normally on 11 eth0
fe80::5054:ff:fe6e:6f67 UDP 123
Dec 16 15:38:47 rac12pri ntpd[5180]: Listening on routing socket on fd #28 for
interface updates
Dec 16 15:38:47 rac12pri ntpd[5180]: 0.0.0.0 c016 06 restart
Dec 16 15:38:47 rac12pri ntpd[5180]: 0.0.0.0 c012 02 freq_set ntpd 0.000 PPM
Dec 16 15:38:47 rac12pri ntpd[5180]: 0.0.0.0 c011 01 freq_not_set
Dec 16 15:38:47 rac12pri systemd[1]: Started Network Time Service.
Dec 16 15:38:48 rac12pri ntpd[5180]: 0.0.0.0 c614 04 freq_mode
[root@rac12pri ~]#
The call to systemctl enable replaces an invocation of chkconfig to automatically start ntpd as
a service (chkconfig ntpd on). Starting the service does not produce any output, hence the need to
check the status.

There is a slight caveat with the use of NTP: it is not the default time keeping service. Another
tool, named chronyd is used instead.

This causes a problem after the next reboot: chronyd will be started, NTPd wont be. The Red
Hat documentation therefore has a section on how to switch:

1 [root@rac12pri ~]# systemctl stop chronyd

2 [root@rac12pri ~]# systemctl disable chronyd
3 [root@rac12pri ~]# systemctl status chronyd

Storage

Shared storage is provided by KVM. I am using my SSDs in the lab from where I create a few
LUNs. These must explicitly be made shareable to be accessible by more than one guest.
Since 12.1.0.2.0 Oracle installs a database for the cluster health monitor by default. Currently I
use the following setup for my lab 12.1.0.2 clusters:

1. +CHM (external redundancy) 1x 15GB

2. +OCR (normal redundancy) 3x 2 GB

3. +DATA (external redundancy) 1 x 15GB

4. +RECO (external redundancy) 1 x 10 GB

If you use the guided installation of Grid Infrastructure the installer will prompt you for a single
disk group only. This means that the CHM database as well as the OCR and voting files be
installed in that disk group. I prefer to separate them though, which is why I create a second disk
group OCR after the installation has completed and move the voting files and OCR out of
+CHM.

DATA and RECO are standard Exadata disk groups and I like to keep things consistent for
myself.

I use fdisk to partition the future ASM disks with 1 partition spanning the whole LUN.

Other tasks

A lot of the other pre-installation tasks can actually be performed during the kickstart
installation. I still like to use SELinux in permissive mode even though-according to
Requirements for Installing Oracle Database 12.1 on RHEL6 or OL6 64-bit (x86-64) (Doc ID
1529864.1)-selinux can be in enforcing. The directive in the kickstart file is

selinux permissive
You shouldnt have to install additional packages-all packages to be installed should go into the
%packages section of the file. Simply copy the package names from the official documentation
and paste below the last package in the section. There is one exception to the rule: cvuqdisk must
be installed from the Oracle installation media.

Settings for /etc/sysctl.conf and /etc/security/limits.conf can also be made in the kickstart file as
shown in the first part of this series.

Storage to be made available to RAC must have permissions set. Since there isnt an ASMLib in
Oracle Linux 7 to my knowledge UDEV will have to be used, and my udev configuration file,
too, is in the first part.

To make sure my user and group IDs for the oracle and grid account are the same I create the
accounts in the kickstart file as well. Passwords are deliberately not set-they may evolve and I
cant possibly remember them all :)

User equivalence can be set up using a technique I have already described in an earlier blog post.
Although the user equivalence setup can be deferred to when you install Grid Infrastructure I still
perform it before to allow me to run the cluster verification tool with the -fixup option.