T EC HN IC AL R E P OR T

Nimble Storage for Oracle 11gR2 RAC on

Oracle Linux 6.4

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 1

Document Revision
Table 1.
1

Date Revision Description

1/11/2013 1.0 Initial Draft

1/14/2013 1.1 Revised

2/25/2014 1.2 Revised

7/31/2014 1.3 Revised iSCSI timeout

11/17/2014 1.4 Updated iSCSI & Multipath

THIS TECHNICAL TIP IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN
TYPOGRAPHICAL ERRORS AND TECHNICAL INACCUURACIES. THE CONTENT IS PROVIDED AS IS,
WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.

Nimble Storage: All rights reserved. Reproduction of this material in any manner whatsoever
without the express written permission of Nimble is strictly prohibited.

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 2

Table of Contents

Introduction ................................................................................................................................................................................. 4

Audience ...................................................................................................................................................................................... 4

Servers Setup for Oracle RAC................................................................................................................................................... 4

Oracle Linux 6.4 Installation ............................................................................................................ 4

Operating system configuration for Oracle ...................................................................................... 5
Oracle Linux Networking Setup ....................................................................................................... 6
Oracle Linux iSCSI Setup ................................................................................................................ 6
Multipath Setup ................................................................................................................................ 7

Nimble Storage Setup ............................................................................................................................................................... 8

Create New Initiator Group .............................................................................................................. 8

Create Volumes and Allow Access ................................................................................................ 10
Protection via Volume Collection .................................................................................................. 15

Oracle 11gR2 RAC Setup........................................................................................................................................................18

Pre-requisites .................................................................................................................................. 18
Oracle ASM Disks Setup ................................................................................................................ 18
Install Grid Infrastructure ............................................................................................................... 20
Install Oracle Software ................................................................................................................... 20

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 3

Introduction
The purpose of this technical white paper is to describe the basic setup of Oracle Linux 6.4 running Oracle RAC 11gR2
with Automatic Storage Management (ASM) when deploying on Nimble Storage.

Oracle performance tuning is beyond the scope of this paper. Please visit www.oracle.com for Oracle Performance
Tuning Guide for more information in tuning your database.

Audience
This guide is intended for Oracle database solution architects, storage engineers, system administrators and IT
managers who analyze, design and maintain a robust database environment on Nimble Storage. It is assumed that the
reader has a working knowledge of iSCSI SAN network design, and basic Nimble Storage operations. Knowledge of
Oracle Linux operating system, Oracle Clusterware, and Oracle database is also required.

Servers Setup for Oracle RAC

Configuration for a 2 node RAC Oracle database.

Oracle Linux 6.4 Installation

During the installation of the operating system, we recommend that Logical Volume Manager (LVM) NOT to be
used for the boot disk. Remove all LVM configurations and use the standard partition method for the boot disk.

/boot partition
SWAP partition
/ partition

[root@racnode1 ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda3 32998256 22234944 9087076 71% /
tmpfs 6291456 2607808 3683648 42% /dev/shm
/dev/sda1 126931 50862 69516 43% /boot

After the installation of the operating system, the following daemons need to be disabled on each node.
o Disabled selinux
o Disabled NetworkManager
o Disable iptables if iptables is required, make sure it is setup correctly.
o Disable ip6tables
If not installed already, the following packages are required for the Oracle 11gR2 and multipathing.

compat-db42-4.2.52-15.el6.x86_64.rpm
compat-db43-4.3.29-15.el6.x86_64.rpm
compat-db-4.6.21-15.el6.x86_64.rpm
cvuqdisk-1.0.9-1.rpm
device-mapper-multipath-0.4.9-64.0.1.el6.x86_64.rpm
device-mapper-multipath-libs-0.4.9-64.0.1.el6.x86_64.rpm

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 4

 ksh-20100621-19.el6.x86_64.rpm
libaio-devel-0.3.107-10.el6.x86_64.rpm
libsysfs-2.1.0-7.el6.x86_64.rpm
libXp-1.0.0-15.1.el6.x86_64.rpm
libXp-devel-1.0.0-15.1.el6.x86_64.rpm
oracleasm-support-2.1.8-1.el6.x86_64.rpm
sysfsutils-2.1.0-7.el6.x86_64.rpm
sysstat-9.0.4-20.el6.x86_64.rpm
unixODBC-2.2.14-12.el6_3.x86_64.rpm
unixODBC-devel-2.2.14-12.el6_3.x86_64.rpm

Operating system configuration for Oracle

Add the below settings to the /etc/systctl.conf file on each node. Run the sysctl p when done.

# For 11g, recommended value for file-max is 6815744

fs.file-max = 6815744
# For 10g, uncomment 'fs.file-max 327679', comment other entries for this parameter and re-
run sysctl -p
# fs.file-max:327679
kernel.msgmni = 2878
kernel.sem = 250 32000 100 142
kernel.shmmni = 4096
net.core.rmem_default = 262144
# For 11g, recommended value for net.core.rmem_max is 4194304
net.core.rmem_max = 4194304
# For 10g, uncomment 'net.core.rmem_max 2097152', comment other entries for this
parameter and re-run sysctl -p
# net.core.rmem_max=2097152
net.core.wmem_default = 262144
# For 11g, recommended value for wmem_max is 1048576
net.core.wmem_max = 1048576
# For 10g, uncomment 'net.core.wmem_max 262144', comment other entries for this
parameter and re-run sysctl -p
# net.core.wmem_max:262144
fs.aio-max-nr = 3145728
# For 11g, recommended value for ip_local_port_range is 9000 65500
net.ipv4.ip_local_port_range = 9000 65500
# For 10g, uncomment 'net.ipv4.ip_local_port_range 1024 65000', comment other entries for
this parameter and re-run sysctl -p
# net.ipv4.ip_local_port_range:1024 65000
# Added min_free_kbytes 50MB to avoid OOM killer on EL4/EL5
vm.min_free_kbytes = 51200

# Nimble Recommended
net.core.wmem_max = 16780000
net.core.rmem_max = 16780000
net.ipv4.tcp_rmem = 10240 87380 16780000
net.ipv4.tcp_wmem = 10240 87380 16780000

Add the settings below to the /etc/security/limits.conf file on each node.

grid soft nofile 131072

grid hard nofile 131072
grid soft nproc 131072

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 5

 grid hard nproc 131072
oracle soft nofile 131072
oracle hard nofile 131072
oracle soft nproc 131072
oracle hard nproc 131072
oracle soft core unlimited
oracle hard core unlimited
oracle soft memlock 50000000
oracle hard memlock 50000000

Add the code below to the /etc/profile file on each node.

if [ $USER = "oracle" ] || [ $USER = "grid" ] ; then

if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi

Oracle Linux Networking Setup

Each node should have four NICs. One is assigned a public IP address. One is assigned a private IP
address. The other two are assigned for iSCSI networks. After assigning the public and private
network, make sure to generate SSH keys on each node and exchange the keys between the 2
nodes for root, grid, and oracle users. After the keys are exchanged, ensure that these users can
login to either node without the need of entering a password. This is required for RAC installation.
SCAN IP addresses are also required in 11gR2 RAC so make sure they are setup in DNS. Also
ensure the /etc/hosts file contains entries of Public, Private, and VIP networks for both nodes.

An example of /etc/hosts file:

# Public
136.18.127.213 racnode1.sedemo.lab racnode1
136.18.127.214 racnode2.sedemo.lab racnode2

# Private
10.18.127.73 racnode1-priv.sedemo.lab racnode1-priv
10.18.127.75 racnode2-priv.sedemo.lab racnode2-priv

# VIP
136.18.127.215 racnode1-vip.sedemo.lab racnode1-vip
136.18.127.216 racnode2-vip.sedemo.lab racnode2-vip

Oracle Linux iSCSI Setup

Leverage the native iSCSI initiator for Oracle. To do this, ensure software iSCSI initiator shown below
has been installed on each node.

iscsi-initiator-utils-6.2.0.872-41.0.1.el6.x86_64

Configure iSCSI as follow on each node:

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 6

 Edit the /etc/iscsi/iscsid.conf and modify the following parameters:
o node.session.timeo.replacement_timeout = 120
o node.conn[0].timeo.noop_out_interval = 5
o node.conn[0].timeo.noop_out_timeout = 10
o node.session.nr_sessions = 4
o node.session.cmds_max = 2048
o node.session.queue_depth = 1024
Make sure the iSCSI NICs (eth1, eth2) have been assigned iSCSI network IP addresses and NET
MASK. Should look something similar to this for both iSCSI NICs.

DEVICE="eth1"
BOOTPROTO=static
HWADDR="00:50:56:AE:30:71"
NM_CONTROLLED="no"
ONBOOT="yes"
TYPE="Ethernet"
UUID="4705a3e9-671a-45a0-8477-8b8b98d56c09"
IPADDR=172.18.128.71
NETMASK=255.255.255.0
MTU=9000

Multipath Setup
Change directory to /etc and using your favorite editor to create a new file called multipath.conf with
the code below

defaults {
user_friendly_names yes
find_multipaths yes
}
devices {
device {
vendor "Nimble"
product "Server"
path_selector "round-robin 0"
path_checker tur
rr_min_io_rq 10
rr_weight priorities
failback immediate
path_grouping_policy group_by_serial
features "1 queue_if_no_path"
}
}

Start multipath by running the following:

o [root@racnode1 ~]# chkconfig multipathd on
o [root@racnode1 ~]# modprobe dm_multipath
o [root@racnode1 ~]# service multipathd start

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 7

Nimble Storage Setup
Creating iSCSI volumes for the Oracle RAC nodes on Nimble Storage is simple.
First, create a new Initiator Group. This Initiator Group contains the Oracle RAC nodes iqn IDs.
Second, create new volumes for the database and allow access to the Initiator Group.

Create New Initiator Group

To create a new initiator group
Open the Nimble Management GUI
Click on Manage
Manage
Manage and select Initiator
Initiator Groups
Groups

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 8

 Click on New
New Initiator Group
Group button and assign a name to the Initiator Group. We recommend
using the Oracle cluster name for distinction.

Click on Add
Add Initiator
Initiator button. Another window will appear. Enter the Initiator Group Name (rac-
cluster in the example below) and the Initiator Name (iqn ID) of both nodes and click OK
OK.
OK You can
find the iqn ID of the Oracle RAC nodes by running the following command on the Oracle Linux
server:

[root@racnode1 ~]# cat /etc/iscsi/initiatorname.iscsi

InitiatorName=iqn.1988-12.com.oracle:496c4aead3e1

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 9

Create Volumes and Allow Access
Click on Manage
Manage
Manage and select Volumes
Volumes
Volumes

Click on New
New Volume
Volume button. Enter a name for the volume. Enter a short description.
Select Limit
Limit access
access radio button and then select Limit
Limit access to iSCSI initiator group
group button and
then click on the drop down arrow to select the initiator group for this cluster (rac-cluster from the
earlier example). Also select the Allow
Allow multiple initiator access
access button then click Next
Next.
Next

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 10

 Enter the size for the volume and click Next
Next.
Next

At the Protection screen, select None

None
None for now and click Finish
Finish.
Finish We will discuss the Protection
option later in the paper.

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 11

Make sure to create enough volumes for the Oracle rac-cluster. Usually in an Oracle RAC ASM environment,
you create one OCR volume for the Clusterware and Voting disk or three for normal redundancy and create
more volumes for the DATA diskgroup and FRA diskgroup depending on how you want to architect your Oracle
database. For the number of volumes for the DATADG and FRADG, please refer to the Oracle Best Practices
Guide.

Once all volumes have been created on the Nimble array, scan for the new volumes on both Oracle RAC
nodes and modify the /etc/multipath.conf on your Oracle Linux servers to use the aliases for the Oracle
volumes. You can modify the multipath.conf file on the first Oracle RAC node only and copy (using scp) the
file to the second Oracle RAC node.

Here is how you can setup multipath for the Oracle volumes.

Scan for disks on the Oracle RAC Linux servers. Note that you need to scan both iSCSI networks. If
there is a single discovery IP address, you can use that IP to run the discovery. With either method,
please ensure that you can see the same volume across two networks.

[root@racnode1 ~]# iscsiadm -m discovery -t st -p <iSCSI IP #1>

[root@racnode1 ~]# iscsiadm -m discovery -t st -p <iSCSI IP #2>
[root@racnode1 ~]# iscsiadm -m node - -login

Note:
Note if you want to login for all volumes that this Linux server has access to, then run the above login
command. If you want to login for a certain volume, then use this command iscsiadm m node T <Target
Volume IQN> - -login.

After the scan completes, you can run the following command to find out which disk serial number
belongs to which LUN so you can alias the volumes properly. You can run this command on the first
Oracle RAC node only. In the below output, sdf and sdg is the one disk (same serial #) which I used
ocr
ocr
ocr as an alias, sdc and sde is one disk which I used racproddata1
racproddata1
racproddata1 as an alias, and so forth.

[root@racnode1 ~]# for a in `cat /proc/partitions | awk '{print $4}' | grep sd`; do echo "### $a: `scsi_id -u
-g /dev/$a`" ; done
### sda:
### sda1:
### sda2:
### sda3:
### sdb: 2bfdb64e06cf5ee696c9ce900abdb3466
### sdd: 2bfdb64e06cf5ee696c9ce900abdb3466
### sdc: 2e4d6149bf39b0c626c9ce900abdb3466
### sde: 2e4d6149bf39b0c626c9ce900abdb3466
### sdg: 2c6c2a2e042cc67d96c9ce900abdb3466
### sdf: 2c6c2a2e042cc67d96c9ce900abdb3466
### sdh: 261dccce8c92d80256c9ce900abdb3466
### sdi: 261dccce8c92d80256c9ce900abdb3466
### sdj: 242d6ac9296637ba66c9ce900abdb3466
### sdk: 242d6ac9296637ba66c9ce900abdb3466

The new /etc/multipath.conf would look something like this. Please refer to Oracle Linux
documentation for specific parameter settings for other storage devices.

defaults {
user_friendly_names yes
find_multipaths yes
}

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 12

devices {
device {
vendor "Nimble"
product "Server"
path_selector "round-robin 0"
path_checker tur
rr_min_io_rq 10
rr_weight priorities
failback immediate
features "1 queue_if_no_path"
path_grouping_policy group_by_serial
}
}
blacklist {
wwid 26353900f02796769
devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
devnode "^hd[a-z]"
}
multipaths {
multipath {
wwid 2c6c2a2e042cc67d96c9ce900abdb3466
alias ocr
}
multipath {
wwid 2e4d6149bf39b0c626c9ce900abdb3466
alias racproddata1
}
multipath {
wwid 261dccce8c92d80256c9ce900abdb3466
alias racproddata2
}
multipath {
wwid 242d6ac9296637ba66c9ce900abdb3466
alias racprodfra1
}
multipath {
wwid 2bfdb64e06cf5ee696c9ce900abdb3466
alias racprodfra2
}
}

Copy the /etc/multipath.conf file to the second Oracle RAC node and reload multipathd daemon on
both Oracle RAC nodes.

[root@racnode1 ~]# /etc/init.d/multipathd reload

Reloading multipathd: [ OK ]

Verify multipath on both Oracle RAC nodes.

[root@racnode1 ~]# multipath -ll

racprodfra2 (2bfdb64e06cf5ee696c9ce900abdb3466) dm-0 Nimble,Server
size=50G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 24:0:0:0 sdd 8:48 active ready running
`- 23:0:0:0 sdb 8:16 active ready running
racprodfra1 (242d6ac9296637ba66c9ce900abdb3466) dm-3 Nimble,Server

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 13

size=50G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 31:0:0:0 sdj 8:144 active ready running
`- 32:0:0:0 sdk 8:160 active ready running
racproddata2 (261dccce8c92d80256c9ce900abdb3466) dm-4 Nimble,Server
size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 29:0:0:0 sdh 8:112 active ready running
`- 30:0:0:0 sdi 8:128 active ready running
racproddata1 (2e4d6149bf39b0c626c9ce900abdb3466) dm-1 Nimble,Server
size=100G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 25:0:0:0 sdc 8:32 active ready running
`- 26:0:0:0 sde 8:64 active ready running
ocr (2c6c2a2e042cc67d96c9ce900abdb3466) dm-2 Nimble,Server
size=5.0G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 27:0:0:0 sdf 8:80 active ready running
`- 28:0:0:0 sdg 8:96 active ready running

Change disk IO scheduler to noop

To set at boot time, add the elevator option at the kernel line in the /etc/grub.conf file:

elevator=noop

To manually set for all disks:

root@mktg04 ~]# multipath -ll | grep sd | awk -F":" '{print $4}' | awk '{print $2}' | while read LUN; do echo
noop > /sys/block/${LUN}/queue/scheduler ; done

Change max_sectors_kb to 1024

To change max_sectors_kb to 1024 for a single volume:

[root@racnode1 ~]# echo 1024 > /sys/block/sd?/queue/max_sectors_kb

Change all volumes:

multipath -ll | grep sd | awk -F":" '{print $4}' | awk '{print $2}' | while read LUN
do
echo 1024 > /sys/block/${LUN}/queue/max_sectors_kb
done

Note:
Note To make this change persistent after reboot, add the commands in /etc/rc.local file.

Change vm dirty writeback and expire to 500 and 3000 respectively

To change vm dirty writeback and expire:

[root@racnode1 ~]# echo 100 > /proc/sys/vm/dirty_writeback_centisecs

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 14

 [root@racnode1 ~]# echo 100 > /proc/sys/vm/dirty_expire_centisecs

Note:
Note To make this change persistent after reboot, add the commands in /etc/rc.local file.

Use performance setting for all available CPUs on the host when possible.

To change CPU governor setting to performance:

[root@racnode1 ~]# echo performance > /sys/devices/system/cpu/<cpu #>/cpufreq/scaling_governor

Change all CPUs

[root@racnode1 ~]# for a in $(ls -ld /sys/devices/system/cpu/cpu[0-9]* | awk '{print $NF}') ; do echo
performance > $a/cpufreq/scaling_governor ; done

Note:
Note To make this change persistent after reboot, add the commands in /etc/rc.local file.

Protection via Volume Collection

Once you have created all volumes for the Oracle rac-cluster, you can manually add the volumes to a Volume
Collection. Volume Collection allows you to take a snapshot via a schedule or ad hoc using the Nimble
Storage management GUI or CLI. In an Oracle environment where multiples volumes have been created for a
particular database, for consistency, you need to take a snapshot of all volumes concurrently. The Volume
Collection allows you to do just that.

After a snapshot has been taken using the Volume Collection feature, the snapshot(s) will be available for
recovery based on your retention policy setting. The snapshot(s) can be deleted manually if you wish to do
so. Please note that you cannot delete a snapshot if there is a clone volume associated with that snapshot.

This particular feature is very useful in an Oracle environment. It allows you to create a clone of your
production database for Test, Development, or Staging environment quickly and easily.

Below are the steps to create a Volume Collection and assign the Oracle volumes to it.

In the Nimble management GUI, click on Manage

Manage,
Manage Protection
Protection,
Protection Volume
Volume Collection
Collection

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 15

 Click on New
New Volume Collection
Collection button

A Create
Create a volume collection
collection window appears. Enter a name for this volume collection and select
Blank
Blank schedule
schedule and click Next
Next.
Next

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 16

 A Synchronization
Synchronization
Synchronization window appears. As of this writing, you can select the None
None
None radio button and
click Next
Next.
Next

A Schedule
Schedule
Schedule window appears. You must have at least one protection (snapshot) schedule defined.
Enter a name and your desired schedule/retention and click Next
Next.
Next

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 17

 A Volume
Volume
Volume window appears. This is where you need to add all of your Oracle volumes to this volume
collection. When done, click Finish
Finish.
Finish

Oracle 11gR2 RAC Setup

Now that the operating system and Nimble storage have been setup for the Oracle rac-cluster, well discuss
installing Grid Infrastructure and Oracle software.

Pre-requisites
Create a local directory on the boot LUN or a separate Nimble LUN for the Grid Infrastructure and
Oracle software on both Oracle RAC nodes. The directories must be exactly the same on both nodes.
Create group ID of oinstall on both servers
Create group ID of dba on both servers
Create user ID of grid on both servers
Create user ID of oracle on both servers

Oracle ASM Disks Setup

o Run /etc/init.d/oracleasm configure command to configure ASM disks on both nodes.

[root@racnode1 ~]# /etc/init.d/oracleasm configure

Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 18

 Default user to own the driver interface []: grid
Default group to own the driver interface []: oinstall
Start Oracle ASM library driver on boot (y/n) []: y
Scan for Oracle ASM disks on boot (y/n) []: y
Writing Oracle ASM library driver configuration: done
Initializing the Oracle ASMLib driver: [ OK ]
Scanning the system for Oracle ASMLib disks: [ OK ]

o Modify the /etc/sysconfig/oracleasm file as follow on both nodes.

# ORACLEASM_ENABELED: 'true' means to load the driver on boot.

ORACLEASM_ENABLED=true
true

# ORACLEASM_UID: Default user owning the /dev/oracleasm mount point.

ORACLEASM_UID=grid
grid

# ORACLEASM_GID: Default group owning the /dev/oracleasm mount point.

ORACLEASM_GID=oinstall
oinstall

# ORACLEASM_SCANBOOT: 'true' means scan for ASM disks on boot.

ORACLEASM_SCANBOOT=true
true

# ORACLEASM_SCANORDER: Matching patterns to order disk scanning

ORACLEASM_SCANORDER="dm
dm"
dm

# ORACLEASM_SCANEXCLUDE: Matching patterns to exclude disks from scan

ORACLEASM_SCANEXCLUDE="sd
sd"
sd

o Make sure ASM module is loaded on both nodes. If not, then load it.

[root@oraclelinux1 ~]# lsmod | grep oracleasm

oracleasm 53352 1

o Create ASM disks on the first node only

[root@racnode1 ~]# /etc/init.d/oracleasm creatdisk OCR /dev/mapper/ocr

[root@ racnode1 ~]# /etc/init.d/oracleasm creatdisk RACPRODDATA1 /dev/mapper/racproddata1
[root@ racnode1 ~]# /etc/init.d/oracleasm creatdisk RACPRODDATA2 /dev/mapper/racproddata2
[root@ racnode1 ~]# /etc/init.d/oracleasm creatdisk RACPRODFRA1 /dev/mapper/racprodfra1
[root@ racnode1 ~]# /etc/init.d/oracleasm creatdisk RACPRODFRA2 /dev/mapper/racprodfra2

o Verify ASM disks

[root@racnode1 ~]# /etc/init.d/oracleasm listdisks

OCR
RACPRODDATA1
RACPRODDATA2
RACPRODFRA1
RACPRODFRA2

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 19

 o Verify ASM disks on the second node

[root@racnode2 ~]# /etc/init.d/oracleasm scandisks

Scanning the system for Oracle ASMLib disks: [ OK ]

[root@racnode2 ~]# /etc/init.d/oracleasm listdisks

OCR
RACPRODDATA1
RACPRODDATA2
RACPRODFRA1
RACPRODFRA2

Install Grid Infrastructure

As grid user, you can now install Grid Infrastructure on the first node only. Follow Oracle documentation on
how to install GI at www.oracle.com. When you get to the part where you have to create ASM disk group for
the Clusterware, change the asm_diskstring to /dev/oracleasm/disks/* and select OK. Remember that
you need to create one disk group for the Clusterware/Vote disk and one or two disk groups for the Oracle
database (i.e. DATADG & FRADG). In this case, for the Clusterware/Vote disk, you can name the disk group
as OCRDG and select only the OCR disk. For the other four disks, create two new disk groups and name
them DATADG and FRADG. For the OCR diskgroup, if you created 3 volumes then use the NORMAL
REDUNDANCY. For other disk groups (i.e DATADG & FRADG), make sure you select EXTERNAL
REDUNDANCY. Please note that the naming and number of disks specified in this document is an example
only. You can have different names and as many disks as youd like.

Install Oracle Software

After the installation of GI, change user to oracle and install Oracle software on the first node only. Again,
follow Oracle documentation on how to install Oracle if you dont already know how. It is much easier to
select the option to install software only first. After the software installation, run dbca to create your
database.

Nimble Storage, Inc.

211 River Oaks Parkway, San Jose, CA 95134
Tel: 877-364-6253) | www.nimblestorage.com | info@nimblestorage.com
2014 Nimble Storage, Inc. Nimble Storage, InfoSight, SmartStack, NimbleConnect, and CASL are trademarks or registered trademarks of
Nimble Storage, Inc. All other trademarks are the property of their respective owners. TR-ORG-1114

TECHNICAL REPORT: NIMBLE STORAGE FOR ORACLE 11GR2 RAC 20