Deploying Oracle12c Rac DB On Xcseries All Flash

Deploying Oracle 12c RAC Database on
Dell EMC XC Series All-Flash

Dell EMC Engineering
January 2017
A Dell EMC Best Practices Guide

Revisions
Date Description
January 2017 Initial release
Acknowledgements
Authors: Chidambara Shashikiran, Henry Wong
The information in this publication is provided “as is.” Dell Inc. makes no representations or warranties of any kind with respect to the information in this
publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.
Use, copying, and distribution of any software described in this publication requires an applicable software license.
Copyright © 2017 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other
trademarks may be the property of their respective owners. Published in the USA [1/19/2017] [Best Practices Guide] [3110-BP-SDS]
Dell EMC believes the information in this document is accurate as of its publication date. The information is subject to change without notice.
2 Deploying Oracle 12c RAC Database on Dell EMC XC Series All-Flash | 3110-BP-SDS
Table of contents
Revisions.............................................................................................................................................................................2
Executive summary.............................................................................................................................................................4
1 Product overview ..........................................................................................................................................................5
1.1 XC Series appliances .........................................................................................................................................5
1.2 XC Series all-flash ..............................................................................................................................................5
1.3 XC Series Acropolis architecture ........................................................................................................................6
1.4 XC Series Acropolis Block Services ...................................................................................................................7
2 Solution infrastructure ..................................................................................................................................................8
2.1 Physical system configuration ............................................................................................................................8
2.1.1 Oracle single instance database configuration ...................................................................................................8
2.1.2 Oracle RAC database configuration .................................................................................................................10
2.1.3 Oracle RAC database configuration using ABS ...............................................................................................11
2.2 XC Series storage and cluster configuration ....................................................................................................12
2.3 Network configuration .......................................................................................................................................12
3 Sizing hypervisor configuration guidelines .................................................................................................................13
3.1 Oracle database VM configuration ...................................................................................................................13
3.1.1 Processor and memory ....................................................................................................................................13
3.1.2 XC Series storage container and VMware storage virtualization .....................................................................13
3.1.3 VM storage controller and virtual disks ............................................................................................................13
3.1.4 VM networking ..................................................................................................................................................15
3.2 VM guest OS configuration for Oracle guidelines ............................................................................................18
3.3 Nutanix CVM.....................................................................................................................................................20
3.4 Storage layout for databases............................................................................................................................21
3.4.1 Oracle ASM for Oracle single instance or RAC ................................................................................................21
3.4.2 File system for Oracle single instance database ..............................................................................................23
3.5 Performance monitoring ...................................................................................................................................23
3.5.1 Nutanix Prism ...................................................................................................................................................24
3.5.2 vSphere client ...................................................................................................................................................25
3.5.3 CVM CLI ...........................................................................................................................................................25
3.5.4 ESX/ESXi CLI ...................................................................................................................................................26
3.5.5 Oracle EM Express ...........................................................................................................................................26
A How to identify and query the disk ids/WWNs ...........................................................................................................27
A.1 How to identify VMware virtual disks in Linux guest.........................................................................................27
A.2 How to identify XC Series volume group volumes in Linux guest ....................................................................28
B Configuration details ...................................................................................................................................................30
C Technical support and resources ...............................................................................................................................31
C.1 Related resources.............................................................................................................................................31
Executive summary
The Dell EMC™ XC Series Web-Scale Hyperconverged appliance powered by Nutanix™ delivers a highly
resilient, converged compute and storage platform that brings benefits of web-scale architecture to business-
critical enterprise applications such as Oracle®.
The XC Series platform is hypervisor agnostic and software installs quickly for deployment of multiple
virtualized workloads. The XC Series Nutanix platform can deliver storage through multiple protocols such as
NFS, SMB, and iSCSI.
This document provides guidelines for design, configuration, and optimization of Oracle single instance and
Real Application Cluster (RAC) databases applications running on XC Series Nutanix infrastructure. The
document also outlines the different storage presentation methods offered by Nutanix to deploy the Oracle
database application.
1 Product overview
1.1 XC Series appliances
The XC Series is a hyperconverged solution that combines storage, compute, networking, and virtualization
into an industry-proven x86 Dell EMC PowerEdge™ server running Nutanix web-scale software. By
combining the hardware resources from each server node into a shared-everything model for simplified
operations, improved agility, and greater flexibility, Dell EMC and Nutanix together deliver simple, cost-
effective solutions for enterprise workloads. Acropolis Distributed Storage Fabric (DSF) delivers a unified pool
of storage from all nodes across the cluster, using techniques including striping, replication, autotiering, error
detection, failover, and automatic recovery.
The XC Series infrastructure is a scale-out cluster of high-performance nodes, or servers, each running a
standard hypervisor and containing processors, memory, and local storage (consisting of SSD flash for high
performance and high-capacity SATA disk drives). Each node runs virtual machines just like a standard
hypervisor host as displayed in Figure 1.
Nutanix node architecture
In addition, the Acropolis DSF virtualizes local storage from all nodes into a unified pool. Acropolis DSF uses
local SSDs and disks from all nodes to store virtual machine data. Virtual machines running on the cluster
write data to ADSF as if they were writing to shared storage.
1.2 XC Series all-flash

Flash technologies are evolving rapidly and flash-based storage is capable of delivering millions of IOPS with
sub-millisecond latency. Also, flash prices have been decreasing, making flash a more cost-effective option
for storage. XC Series appliances are now available in both hybrid and all-flash variants. The blazing
performance, coupled with integrated data efficiency, resiliency and protection capabilities on all-flash variants
makes them the preferred choice for enterprise applications such as Oracle which require higher IOPS and
low latency.
The solution described in this document used a 3-node XC630-10AF all-flash cluster comprising of ten 800GB
SSDs in each node.
1.3 XC Series Acropolis architecture
Nutanix software provides a hyperconverged platform that uses Acropolis Distributed Storage Fabric (DSF) to
share and present local storage to all the virtual machines in the cluster. The general XC Series Nutanix
architecture is shown in Figure 2.
Nutanix architecture
Acropolis DSF virtualizes the storage across all nodes and presents the same to the hypervisor as one large
pool of shared storage. The DSF replicates writes synchronously to at least one remote XC Nutanix node to
ensure cluster resiliency and availability. Local storage for each XC Nutanix node in the architecture is
presented as one large pool of shared storage to hypervisor.
Each node runs an industry-standard hypervisor — VMware® ESXi®, Microsoft® Hyper-V®, or Acropolis
Hypervisor (AHV) — and the Nutanix Controller VM (CVM). The Nutanix CVM runs the Nutanix software and
serves I/O operations for the hypervisor and all VMs running on that host. Each CVM connects directly to the
local storage controller and its associated disks thereby reducing the storage I/O latency. The data locality
feature ensures virtual machine I/Os are always served by the local CVM on the same hypervisor node,
improving the VM I/O performance regardless of where it runs.
1.4 XC Series Acropolis Block Services
A feature called Acropolis Block Services (ABS) was released with Acropolis OS 4.7 (AOS). It allows DFS
resources to be exposed directly to a virtualized guest OS or physical hosts using the iSCSI protocol. This
capability enables support for several use cases such as shared storage for Oracle RAC and other
applications that require shared storage.
The XC Series Nutanix storage configuration for ABS is handled through a construct called a volume group
(VG). A VG is a collection of volumes commonly known as virtual disks (vdisks). ABS presents these vdisks to
virtual machines and physical servers using iSCSI protocol. Multiple hosts can share the vdisks associated
with a VG as shown in Figure 3. This is very helpful for shared storage use cases such as Oracle RAC or
Windows server clustering.
XC Nutanix Acropolis Block Services architecture
2 Solution infrastructure
The configuration and solution components are described in this section.
2.1 Physical system configuration
2.1.1 Oracle single instance database configuration

Oracle single-instance database applications can be deployed on virtual machines and can be scaled easily
by adding additional virtualized database instances.
The physical configuration for this environment starts with the basic, three-node XC Series cluster shown in
Figure 4. Single-instance databases can be deployed on VMs on each host of the cluster as shown in the
figure. This architecture enables linear scaling of capacity and performance as you increase number of nodes.
Oracle single-instance configuration
As shown in Figure 4, the local storage controller on each host ensures that storage performance as well as
storage capacity increases when additional nodes are added to the XC Series. Each CVM is directly
connected to the local storage controller and its associated disks. By using local storage controllers on each
ESXi host, access to data through Acropolis DSF is localized. It does not require data to be transferred over
the network, thereby improving latency.
Oracle Automatic Storage Management (ASM) is highly recommended for database-related files. The vdisk
presented to each VM is mapped as an Oracle ASM disk and ASM disk groups are carved out using the ASM
disks. The ASM disk groups are also shown in Figure 4.
2.1.2 Oracle RAC database configuration
Oracle RAC allows running multiple database instances on multiple servers in the cluster against a single
database. The database spans multiple servers but appears as a single unified database to end-user
applications. This architecture helps provide the highest availability and reliability to the Oracle database
applications.
The architecture in Figure 5 is similar to the previous configuration, but the three VMs on each host are
grouped together to form an Oracle RAC.
Oracle RAC configuration
2.1.3 Oracle RAC database configuration using ABS
The XC Series Nutanix ABS feature enables DSF storage resources to be presented directly to VMs and
physical servers using iSCSI. This configuration is similar to the previous configuration, but the database files
are presented to the VMs using iSCSI. A volume group consisting of multiple vdisks is created to store
database-related files and the vdisks are presented to VMs using iSCSI as shown in Figure 6.
Oracle RAC configuration using ABS
2.2 XC Series storage and cluster configuration
Each XC430 Series node used in this configuration is comprised of the following hardware components:
 Ten 800GB SATA SSDs
 Two 16-core Intel® Xeon® E5-2630 v3 2.40GHz processors
 Twelve 16GB DDR-4 QR 2133MHz RAM modules (192GB total)
The minimum number of XC Series nodes in a cluster is three. When clustered together, the storage across
three nodes is virtualized together to create a single storage pool, and one or more containers can be created
on top of the storage pool. The storage container is presented to all nodes as shared storage within the
cluster.
The cluster attempts to keep virtual machines and their associated storage on the same cluster node for
performance consistency. However, each cluster node is connected to, and communicates with, the other
nodes on a 10Gb network. This communication allows virtual machines and their associated storage to reside
on different cluster nodes.
2.3 Network configuration

Figure 7 shows the ideal network topology used for this solution. Two Dell EMC Networking S4810 switches
are stacked together to provide high availability. It is recommended to use at least two top-of-rack switches for
redundancy. Redundancy across the two switches is provided using the LAG, or stack, connection.
Network configuration
3 Sizing hypervisor configuration guidelines
3.1 Oracle database VM configuration
3.1.1 Processor and memory

Sizing the vCPUs and memory of the virtual machines appropriately requires understanding the Oracle
workload. Avoid overcommitting processor and memory resources on the physical node. Use as few vCPUs
as possible because performance might be adversely impacted when using excess vCPUs due to the
scheduling constraints.
Hyperthreading is a hardware technology on Intel processors that enables a physical processor core to act
like two processors. In general, there is a performance advantage to enabling hyperthreading on the newer
Intel processors.
Each VMware vSphere® physical node also runs a Nutanix CVM. Therefore, consider the resources required
for the CVMs. Only one CVM would be running on a physical node, and it does not move to another physical
node when a failure event occurs.
While it is possible to support multiple Oracle database VMs on a same physical node, for performance
reasons, it is better to spread them out on multiple nodes and minimize the number of database instances
running on the same node. In the case of Oracle RAC, the RAC-instance VMs should run on different physical
nodes. VM-host affinity or anti-affinity rules can be set up for database VMs to define where they can run
within the cluster.
3.1.2 XC Series storage container and VMware storage virtualization

On XC Series storage, it is typical to have a single storage container comprised of all SSDs and HDDs in the
cluster so that it can maximize the storage capacity and manage the auto-tiering more efficiently. The single
container is mounted on the ESX hosts through NFS as a datastore. Multiple virtual disks are created from
the same datastore and presented to the guest OS as SCSI disks that can be used by Oracle ASM.
Nutanix ABS allows storage to be presented directly into a non-VM physical host or virtualized guest OS
through iSCSI, bypassing the VMware storage virtualization layer. Additional consideration and extra
configuration steps are required for both Nutanix and the guest OS.
For virtual machines on VMware, it is recommended to present storage as virtual disks with VMware storage
virtualization because it offers a good balance between flexibility, performance, and ease of use.
Find more information on ABS at the Nutanix portal.
3.1.3 VM storage controller and virtual disks

Typically, an Oracle database spans across multiple LUNs to increase performance by allowing parallel I/O
streams. In a virtualized environment, multiple virtual disks are used instead. Nutanix recommends to have at
least four to six database virtual disks and add more disks, depending on the capacity requirements, to
achieve better performance.
It is best practice to create multiple controllers and separate guest OS virtual disk from database virtual disks.
The guest OS virtual disk should be on the primary controller. Additional controllers are created to separate
the virtual disks for data and log files.
Table 1 shows an example configuration of controllers and virtual disks for an Oracle database VM.
Virtual adapter and vdisk configuration for Oracle database VM
Controller Adapter type Usage Virtual disk
Controller 0 LSI Logic Parallel Guest OS 1 x 100GB
Controller 1 Paravirtual Data files, redo logs 4 x 200GB
Controller 2 Paravirtual Archived logs 2 x 200GB
Use the default adapter type LSI Logic Parallel for SCSI controller 0.
Choose Paravirtual SCSI (PVSCSI) adapter type for controllers where virtual disks are used for data files,
redo logs, and archived logs. The PVSCSI adapter allows greater I/O throughput and lower CPU utilization.
VMware recommends using this for virtual machines with demanding I/O workloads.
3.1.3.1 Shared access for virtual disks

By default, VMware does not allow multiple virtual machines to access the same virtual disks. In an Oracle
RAC implementation where multiple database VMs need to access the same set of virtual disks, the default
protection must be disabled by setting the multi-writer option. The option can be found in the virtual machine
setting in the vSphere web client. It can be set when new virtual disks are created or when the virtual machine
is powered down.
3.1.3.2 Queue depth and outstanding disk requests
Splitting virtual disks across multiple controllers increases the limits of outstanding I/Os which a virtual
machine supports. For a demanding I/O workload environment, the default queue depth values might not be
sufficient. The default PVSCSI queue depth is 64 per virtual disk and 254 per virtual controller. To increase
these settings, refer to the VMware KB article 2053145.
For vSphere versions prior to 5.5, the maximum number of outstanding disk requests for virtual machines
sharing a datastore/LUN is limited by the Disk.SchedNumReqOutstanding parameter. Beginning with
vSphere version 5.5, this parameter is deprecated and is set per LUN. Review and increase the setting if
necessary. Refer to the VMWare KB article 1268 for details.
3.1.3.3 Enabling virtual disk UUID

It is important to correctly identify the virtual disks inside the guest OS before performing any disk-related
operations such as formatting or partitioning the disks. WWN is commonly used as the unique identifier to
identify disks that support it. For the guest OS to properly see this information of the virtual disks, the
EnableUUID parameter must be set to TRUE in the virtual machine configuration. The steps to set this
parameter can be found in appendix A.
3.1.4 VM networking
A minimum of two 10GbE interfaces are recommended for each ESXi host. The actual number required
depends on how many vSwitches and the total network bandwidth requirement. Each host should connect to
dual redundant switches for network path redundancy as described in section 2.3. Table 2 shows number of
vSwitches and their target usage.
vSwitches and target use

Physical
vSwitch Speed Network Usage
adapters
vSwitchNutanix NA Intra CVM and svm-iscsi-pg Primary storage

ESXi host vmk-svm-iscsi-pg communication path
vSwitch0 vmnic0 10GB Management network, Management traffic, VM

vmnic1 VM network public traffic, inter-node
communication
vSwitch1 vmnic2 10GB VM network: Oracle Private Oracle RAC

vmnic3 interconnect
vSwitch2 vmnic4 10GB VM network: iSCSI Dedicated iSCSI traffic to

vmnic5 VMs
3.1.4.1 Nutanix local virtual switch
Each ESXi host which is part of the cluster has a local vSwitch automatically created as shown in the
following image.
This switch is used for local communication between the CVM and ESXi host. The host has a vmkernel
interface on this vSwitch and the CVM has an interface bound to a port group called svm-iscsi-pg which
serves as the primary communication path to the storage. This switch is created automatically when the
Nutanix operating system is installed. It is recommended not to modify this virtual switch configuration.
3.1.4.2 Management/inter-node network

The virtual switch was created using two 10GbE physical adapters, vmnic0 and vmnic1, as shown in the
following image. The management traffic is very minimal but Acropolis DSF needs connectivity across CVMs.
This connectivity is used for synchronous write replication and also Nutanix cluster management. This
network is used for VMware vSphere vMotion® and other cluster management activities as well.
3.1.4.3 Public VM network

All public client access to the virtual machines flow through this network. The client traffic can come in bursts
and sometimes the largest amount of data might be transferred between client and VMs. It is recommended
to set this up on the 10Gb network.
3.1.4.4 Oracle RAC interconnect
When deploying Oracle RAC, Oracle recommends a setting up a dedicated network for inter-RAC node
traffic. A separate vSwitch can be set up with redundant physical adapters to provide a dedicated RAC
interconnect network. Only RAC traffic should go on this network.
3.1.4.5 iSCSI network

Typically, iSCSI is not a normal protocol used in an XC Series Nutanix environment. The ABS feature, part of
Nutanix release 4.7, enables presenting DSF storage resources directly to virtualized guest operating
systems or physical hosts using the iSCSI protocol. If this feature is used, it is recommended to use at least
two 10GbE interfaces dedicated for iSCSI traffic. The vSwitch configuration used for iSCSI connectivity in the
solution is shown in the following image.
3.1.4.6 Other networking best practices
Additional networking best practices include the following:
 It is recommended to use dedicated NICs on the hosts for management, iSCSI, and RAC-
interconnect traffic. Also, dedicated VLANs are recommended to segregate each type of traffic.
 For a standard virtual switch configuration, the default load-balancing policy is recommended: Route
based on originating virtual port.
These attributes help simplify implementation for configurations such as LACP.
 The standard network packet size is 1500 MTU. Jumbo frames send network packets in a much
larger size of 9000 MTU. Increasing the transfer unit size allows more data to be transferred in a
single packet which results in higher throughput, and lower CPU utilization and overhead. Use Jumbo
frames only when all the network devices — including the network switches, CVMs, VMs, and ESXi
hosts — on the network path can support the same MTU size.
3.2 VM guest OS configuration for Oracle guidelines

Oracle and VMware support most of the mainstream Linux® distributions. Dell recommends Oracle Linux or
Red Hat® Enterprise Linux for running Oracle databases due to the wide customer install base and the strong
supports of the database products by these companies. However, customers might choose any supported
Linux distribution based on their preferences. For this document, the information is based on Oracle Linux.
VMware recommends using the PVSCSI and VMXNET3 drivers for greater performance capability. Many
mainstream Linux distributions might have these drivers included and installed by default. However, in order
to ensure the drivers are at the latest version, obtain the latest version of VMware tools and install them in the
guest OS. Refer to the VMware KB article 1014294 for more information about general VMware tools
installation instructions.
Installing and configuring the OS for an Oracle database in a virtual machine is similar to doing so on a
physical host.
1. Install the base OS from a virtual CD or ISO.

2. Update the guest OS, software packages, and bug fixes to the latest version from the vendor’s yum
repository.
3. Install the latest version of VMware tools.
4. Configure the guest OS to synchronize time from a trusted server. This is particularly important if
Oracle RAC will be used. All RAC virtual machines must maintain a synchronized time across the
RAC cluster. It is recommended to use NTP on Oracle Linux 6.x/Red Hat Enterprise Linux 6.x or
chronyd on Oracle Linux 7.x/Red Hat Enterprise Linux 7.x.
5. Transfer the Oracle installation media to the virtual machines. If deploying multiple instances, it might
be beneficial to set up an NFS mount and share the common media through NFS.
6. Disable the NetworkManager service.
7. Disable the avahi_daemon service if Oracle RAC will be deployed on the virtual machine.
8. Oracle recommends tuning the swapping priority on database servers. Set vm.swappiness=5 in
/etc/sysctl.conf.
9. Review the 12c Oracle Database Preinstallation Tasks and 12c Oracle Grid Infrastructure Installation
Checklist. Other Oracle versions can be found on https://docs.oracle.com.
10. Create Oracle OS users and groups (user oracle with primary group of oinstall and user grid with
primary group of oinstall).
11. Present virtual disks to the guest OS in vSphere.
12. Identify virtual disks using the procedures outlined in appendix A.
13. VMware recommends setting the disk timeout value to 180 seconds. This is typically handled by the
VMware tools where a udev rule is created to set this timeout. It has been discovered that this udev
rule is missing in the VMware tools for Oracle Linux 7.x/RHEL 7.x systems. To work around this
issue, the file can be created manually in /etc/udev/rules.d/99-vmware-scsi-timeout.rules with the
following contents:
ACTION=="add", SUBSYSTEMS=="scsi", ATTRS{vendor}=="VMware ",

ATTRS{model}=="Virtual disk", RUN+="/bin/sh -c 'echo 180
>/sys$DEVPATH/timeout'"
14. Reboot the system or run the following command to apply the rule without rebooting:
udevadm trigger --action=add
15. Verify the new setting:
lsscsi –l or
cat /sys/block/sdX/device/timeout
16. Assign ownership and permission to the virtual disks for Oracle and make the setting persistent
across restart. This can be configured using the Linux udev facility. See section 3.4.1 for more
information on configuring Oracle ASM and an udev example.
17. For Oracle Linux, install the Oracle preconfiguration script from Oracle’s yum repository. The package
automates some of the Oracle preinstall configuration steps such as adjusting kernel parameters,
configuring the oracle users OS limits, verifying and installing pre-requite packages.
a. Enable ol7-addon or ol6-addon in /etc/yum.repos.d/public-yum-olX.repo:
yum list oracle-rdbms-servers*
oracle-rdbms-servers-12cR1-preinstall
oracle-rdbms-servers-11gR2-preinstall
b. Install the corresponding package version using yum install command.

c. For other Linux distributions, follow the 12c Oracle Database Preinstallation Tasks and 12c
Oracle Grid Infrastructure Installation Checklist. Other Oracle versions can be found on
https://docs.oracle.com.
18. When using virtual disks with VMware storage virtualization, Multipath IO (MPIO) is not required
because each disk will only have a single path in the guest OS.
19. If Nutanix ABS is configured and virtual disks are presented directly to the guest OS through iSCSI,
the Linux iSCSI software initiator must be configured. MPIO with iSCSI disks is no longer required
starting with Acropolis OS 4.7 because it accomplishes session resiliency and HA with the new
external application service IP and iSCSI redirection. For versions prior to 4.7, MPIO with iSCSI disks
are necessary.
20. Nutanix recommends increasing the iSCSI MaxTransferLength to 1MB for large block sequence
workloads with I/O sizes of 1MB or larger. This setting can be changed in /etc/iscsi/iscsid.conf.
node.conn[0].iscsi.MaxRecvDataSegmentLength = 1048576
Find more information on Acropolis Block Services and client iSCSI configuration for use with ABS at the
Nutanix portal.
3.3 Nutanix CVM

The Nutanix CVMs are virtual machines running on vSphere. They have direct control of the SSDs and HDDs
on the ESXi hosts through VMDirectPath I/O. They come preconfigured with modest settings so that
customers can start using the storage platform without much startup time. However, for heavy workload
applications such as databases, special attention should be paid to the CVM resources since they serve as
the primary access points to the SSDs and HDDs. During internal database workload testing, the CVMs
processor and memory resources were strained and limited the I/O performance. Dell EMC recommends
increasing each CVM vCPU to a minimum of 12 and memory to 32GB when heavy I/O workloads are
planned.
Continue to monitor the performance of CVMs periodically to fine-tune the configuration to achieve the
optimal performance level. Section 3.5 describes several ways to monitor the performance of CVM and other
infrastructure components.
3.4 Storage layout for databases
This section discusses common storage practices for Oracle databases based on Oracle database 12c.
3.4.1 Oracle ASM for Oracle single instance or RAC

Oracle ASM is the preferred storage management solution for either single instance database or RAC. Oracle
ASM takes place of the traditional Linux volume manager and file system. When proper ownership and
permission are set on the disks, ASM takes over the management of the disks and create disk groups where
data files reside and are managed automatically. Using Oracle ASM also improves overall performance as it
stripes all files across all disks evenly. Another advantage of ASM is that disks can be added or removed
dynamically without any downtime, and files are restriped across the remaining disks automatically to ensure
optimal performance. ASM also has many advantages over the Logical Volume Manager (LVM) file-system
solution (discussed in section 3.4.2) and it should be considered as the preferred choice whenever possible.
Some of the benefits of ASM include:
 Automatic file management

 Online data files rebalance across disk volumes
 Online addition and removal of disk volumes
 Single solution for both volume and file management
The most common way to consume XC Series storage is to mount it as a datastore in the VMware hypervisor
through NFS. Virtual disks are then created through the regular vSphere client and added to the Linux guest
OS virtual machines. Oracle ASM can take control of these virtual disks just like any physical disks. Thus, no
extra configuration steps are required to make XC Series storage work with ASM.
The following Oracle ASM best practices are recommended:
 Use Linux device management facility, udev, to assign ownership and access permissions
persistently on disk volumes. Without proper permission, ASM cannot manage and control the disk
volumes. For example, the udev rule specifies a disk volume with the device id/WWN,
36000c294a883a308becef0cd8448725a, creates a symlink in /dev/asmdatadg1, and assigns
grid:oinstall and 0660 permission to the device file.
/etc/udev/rules.d/99-oracle-asm.rules:
KERNEL=="sd*", PROGRAM="/lib/udev/scsi_id --page=0x83 --whitelisted --

device=/dev/%k",RESULT=="36000c294a883a308becef0cd8448725a",
SYMLINK+="asmdatadg1", OWNER:="grid", GROUP:="oinstall",MODE="0660"
To find out how to query the disk id/WWN, refer to appendix A.
 Oracle supports asynchronous I/O by default. The disk_asynch_io database parameter is set to
true. Oracle recommends using the deadline I/O scheduler for Oracle ASM. Oracle UEK kernels
have the deadline I/O scheduler as default. The Nutanix recommendation is to use the deadline I/O
scheduler for hybrid configurations consisting of SSDs and HDDs. If the default disk I/O scheduler is
not deadline, a udev rule file can be created to set it. The following shows an example of the rule.
60-disk-scheduler.rules:
ACTION=="add|change", KERNEL=="sd*", RUN+="/bin/sh -c '/bin/echo deadline

> /sys$env{DEVPATH}/queue/scheduler'"
For all-flash configurations, Nutanix recommends the NOOP I/O scheduler because it can reduce I/O
latency, improve throughput, and help reduce CPU cycles. The following shows an example of the
udev rule to set IO scheduler as NOOP.
60-disk-scheduler.rules:
ACTION=="add|change", KERNEL=="sd*", RUN+="/bin/sh -c '/bin/echo noop >

/sys$env{DEVPATH}/queue/scheduler'
 Although ASM supports volumes of different sizes in the same disk group, to ensure best storage
performance, all volumes should have the same space capacity and performance characteristics.
 A minimal of two disk groups should be created: one for data files and one for a fast recovery area.
 If Oracle Grid Infrastructure will be installed for Oracle RAC, create a separate disk group to hold the
crs/voting disks.
 Oracle ASM offers several levels of data redundancy to increase data availability. In High
Redundancy, ASM creates two mirror copies, and in Normal Redundancy mode, ASM creates one
mirror copy. In External Redundancy mode, there is only one copy of the data. Since XC Series
storage already offers data protection and HA by creating two replicas of the data on the storage level
by default, it is sufficient to choose External Redundancy for the ASM disk groups. However,
customers can choose other redundancy options to increase the level of protection, but must make
sure to factor in the additional storage required by using one of these redundancy modes.
 It is recommended to keep the ASM allocation unit size as 1MB which is the default value for ASM
disk groups.
Table 3 shows an example of ASM disk group configuration:
Example ASM disk group configuration

ASM disk Number of
AU size Description
group virtual disks
DATA 4 1 MB Database files; temporary table space; online redo

logs; system related table spaces such as SYSTEM
and UNDO
FRA 2 1 MB Archive logs and backup data
CRS 1 1 MB Clusterware-related information such as the OCR

and voting disks
3.4.2 File system for Oracle single instance database
Oracle also supports storing data files on modern file systems such as ext3/ext4. xfs is a high-performance
file system many Linux distributions include as default option. However, as of this time, Oracle has not
certified or tested Oracle databases running on xfs. Therefore, it should not be used for storing data and log
files. For performance reasons, Linux LVM should be considered for any file systems that hold the database
files. LVM provides the capability to distribute files across multiple disks similar to Oracle ASM. The use of file
system with LVM is recommended for a single-instance configuration only. If shared disks are required such
as Oracle RAC, use Oracle ASM instead. The following guidelines should be followed:
 Nutanix recommends that a volume group consist of at least four to six volumes; more volumes can
be added later based on the capacity requirements.
 All volumes should have the same space capacity and performance characteristics. The maximum
size available to the database is restrained by the smallest volume multiplied by the number of
volumes in the volume group.
 Create two volume groups and one logical volume in each volume group: one for database files and
one for archived logs.
 Striping the logical volumes can significantly improve the I/O performance.
 Create an ext4 file system on the logical volumes. Use of ext4 file system is recommended at this
time. For a complete list of supported file systems, refer to Oracle KB article 1601759.1.
 Oracle supports asynchronous I/O by default. The disk_asynch_io database parameter is set to
true.
 Set the database parameter filesystemio_options = setall to instruct Oracle to use asynchronous
I/O and direct I/O when the underlying file system supports this.
 Using Oracle Managed Files is recommended to simply the administration and management of
database files on file systems.
 The Oracle software binaries should be installed into a separate file system, separated from the root
file system and the database file systems.
Table 4 shows an example of LVM configuration.
Example LVM configuration

Volume Number of Logical
Striped Description
group virtual disks volume/FS
DATA 4 /data Y Database files; temporary table space; online

redo logs; system related table spaces such as
SYSTEM and UNDO
FRA 2 /fra Y Archive logs and backup data
3.5 Performance monitoring

It is important to monitor the performance of various infrastructure components such as CVM, Oracle VM,
CPU, memory, storage, and the network. The following subsections describe the different ways to find this
information. One or multiple methods can be combined to provide a complete picture of the infrastructure
performance.
3.5.1 Nutanix Prism
Prism is a rich graphical HTML-based management console for configuring, managing, and monitoring all
components within the infrastructure. It provides excellent performance monitoring and analytic capabilities. It
also gathers certain data from the VMware vSphere and display all relevant information through a single
Prism interface. For example, to see the virtual machines, including CVMs and performance data, click VM >
Table > Check Include Controller VMs.
The Analysis section in Prism allows creating custom charts for almost every aspect of Nutanix platform and
some VMware vSphere metrics, providing invaluable insight into the environment. Customers can easily see
the current and historical trends by sliding the time scale at the top of the screen. Charts can be added from
the existing charts in other pages or can be easily created using a simple popup where components are
selected. Here are some of the useful charts:
 Cluster hypervisor IOPS
 Cluster hypervisor IO latency
 Cluster IO bandwidth
 Cluster read/write IOPS
 Controller IOPS
 Controller latency
 Controller bandwidth
 Container IOPS
 Container latency
 Container bandwidth
 Disk IOPS
 Cluster CPU/memory utilization
 Content cache utilization
3.5.2 vSphere client
While Prism can see deep into the distributed storage platform, VMware vSphere provides extensive
information on the hypervisor and the VMs. Since CVMs are just other virtual machines managed by vSphere,
many resource statistics can be obtained through the regular vSphere client. To view this information, click
vSphere Web Client > Physical node > VM > Monitor tab > Performance.
3.5.3 CVM CLI

The other option to obtain the CVM performance data is to connect directly to each CVM through ssh and run
the non-graphical command-line utilities:
1. ssh to the individual CVM IP addresses and log in with the Nutanix administration account (typically
nutanix).
2. Run the top utility to show real-time system resources such as processor, memory, and swap.
3. Run the free utility to show real-time memory and swap consumption.
4. Run additional Linux performance utilities as desired, such as vmstat, mpstat, and iostat.
The downside of the CLI is that the utilities offer only real-time information. To see historical data and trends,
use vSphere client or Prism.
3.5.4 ESX/ESXi CLI
To see performance of a specific ESX host, connect directly to the ESX host and run the esxtop command
utility which provides real-time resource utilization:
1. ssh to the ESX host.

2. Log in with the administrator account.
3. Run the esxtop command.
The information pertains only to that specific host and does not include any cluster-wide information. esxtop
breaks down the physical CPU utilization which can be very useful for troubleshooting performance issues.
3.5.5 Oracle EM Express

Oracle provides a free web-based tool, Enterprise Manager Database Express (EM Express), for managing
Oracle Database 12c. EM Express is included with the database and does not require any separate middle-
tier application layer so it can be run directly on the database VM.
EM Express offers a friendly web UI. Under the performance hub, customers can find performance data
collected by the database in real-time or from a given period of time in the past. Some useful performance
data includes:
 Database resource utilization

 Top SQL sessions
 SQL monitor
 Workload characteristics
 Top wait class
 Redo logs throughput
 IOPS
 IO throughput
 IO latency
 ADDM
A How to identify and query the disk ids/WWNs
Correctly identifying the database disks and setting the appropriate ownership and permission on them are
critical steps to take to ensure Oracle can properly take control of the disks. One of the most reliable methods
to identify the correct disks to use is to query the disk ids or WWNs. In the Linux guest, the scsi_id command
is commonly used to query the information. On the storage platform, depending on where the disks are
created (either in VMware or in ABS), the information is extracted from either VMware or the CVM. The
following subsections describe the procedures to query the disk information.
A.1 How to identify VMware virtual disks in Linux guest

Perform the following steps:
1. Set the EnableUUID parameter to TRUE in the virtual machine configuration. Without it set, the Linux
command scsi_id would not return the proper information. Click VM > Edit Settings > VM Options >
Advanced > Configuration Parameters > Add row.
2. Reboot the VM.
3. On the ESXi console, navigate to the VM directory and extract UUIDs from the vmdk files:
[root@XCORANODE1:/]# cd /vmfs/volumes/50eb5287-4ebf3591/oracle-node10
[root@XCORANODE1:/]# ls *vmdk|egrep -v flat|while read file;do echo -n
"$file" : ;awk -F= '/uuid/ {print $NF}' "$file"|sed -e 's/[ |-]//g' ; done
oracle-node10.vmdk :"6000C29336f6899c7fa6abfecaeea954"
crsvote.vmdk :"6000C298843d8fd2b37a500e0ad1ec66"
datadg1.vmdk :"6000C294a883a308becef0cd8448725a"
datadg2.vmdk :"6000C29e2e7074291cbca476ad8cf554"
datadg3.vmdk :"6000C29768dd2bc0ca3af87b113a7b08"
fradg1.vmdk :"6000C290368b65bc7fa1bcbb8d7a4bd4"
fradg2.vmdk :"6000C29f55ee3b293f26e7105f51372d"
oracle-node10.vmdk :"6000C29cacd0ff627d86bd3489c6c5e3"
4. On Linux guest (Red Hat/Oracle Linux 6 or 7), extract the UUID with scsi_id commands:
[root@oracle-node10 ~]# ls -1 /dev/sd?|while read device; do echo -n

$device :; /lib/udev/scsi_id --page=0x83 --whitelisted --device=$device;
done
/dev/sda :36000c290368b65bc7fa1bcbb8d7a4bd4
/dev/sdb :36000c29f55ee3b293f26e7105f51372d
/dev/sdc :36000c29336f6899c7fa6abfecaeea954
/dev/sdd :36000c29cacd0ff627d86bd3489c6c5e3
/dev/sde :36000c298843d8fd2b37a500e0ad1ec66
/dev/sdf :36000c294a883a308becef0cd8448725a
/dev/sdg :36000c29e2e7074291cbca476ad8cf554
/dev/sdh :36000c29768dd2bc0ca3af87b113a7b08
5. Match up results from steps 3 and 4 to show the vmdk and Linux device file relationship.
A.2 How to identify XC Series volume group volumes in Linux guest

ABS exposes the Nutanix DSF through the iSCSI protocol directly into the virtualized guests or physical
systems. On the virtualized guests or physical systems, iSCSI services would be required to connect to ABS.
Perform the following steps:
1. Connect to one of the CVMs.

2. Invoke the ncli interface.
3. Issue the volume-group list command. A list of volume groups and the volumes within each volume
group would be shown.
Look for the Disks attribute in the output. All the virtual disks and their attributes including the VM
Disk UUID would be listed.
ncli> volume-group list
Attachment List : [{VM UUID=null, iSCSI Initiator=iqn.1988-

12.com.oracle:a5101ccc36b2}, {VM UUID=null, iSCSI Initiator=iqn.1988-
12.com.oracle:9e4ff9ee93e5}, {VM UUID=null, i
SCSI Initiator=iqn.1988-12.com.oracle:32bfb29df95}]
Description :
iSCSI Target : oraclevg2-160fbe38-8f97-4e5f-b7a8-
dcad2cdb89e4
Disks : [{VM Disk Size=100 GiB (107,374,182,400
bytes), VM Disk UUID=cfced470-276b-4b1e-948a-133c8078409c, Index=0,
Container ID=3222}, {VM Disk Size=100 GiB (107,
374,182,400 bytes), VM Disk UUID=45a756e8-e3ff-411c-a354-e29beed44235,
Index=1, Container ID=3222}, {VM Disk Size=100 GiB (107,374,182,400
bytes), VM Disk UUID=27613917-b45f-43d2-96c5-61d
e3bafcbba, Index=2, Container ID=3222}, {VM Disk Size=100 GiB
(107,374,182,400 bytes), VM Disk UUID=6bd0060d-d37b-4cb2-8ce4-
fc669c4f3fc3, Index=3, Container ID=3222}, {VM Disk Size=100 Gi
B (107,374,182,400 bytes), VM Disk UUID=71a592b5-5555-41ea-b27e-
d9ec37d3c84e, Index=4, Container ID=3222}, {VM Disk Size=100 GiB
(107,374,182,400 bytes), VM Disk UUID=7109ed82-4ea7-437c-b
2f3-ca096671437f, Index=5, Container ID=3222}]
UUID : 160fbe38-8f97-4e5f-b7a8-dcad2cdb89e4
Shared : true
Name : ORACLEVG2
4. On Linux guest (Red Hat/Oracle Linux 6 or 7), extract the UUID with scsi_id commands:
[root@oracle-node10 ~]# ls -1 /dev/sd?|while read device; do echo -n

$device :; /lib/udev/scsi_id --page=0x83 --whitelisted --device=$device;
done
5. Match up results from steps 3 and 4 to show the vmdk and Linux device file relationship.
B Configuration details
Solution configuration: hardware components Description
XC Series Nutanix 3 x XC630-10AF: A single VM will be hosted on

cluster  ESXi 6.0 each server and Oracle RAC
 AOS: 4.7.2 database will be configured using
 2 x Intel Xeon Processor E5-2698 v3 @ the three VMs.
2.30GHz
 512 GB RAM (16 x 32GB DIMMs)
 10 x 800GB SATA SSD drives
 1 x Intel(R) X520/2P I350 rNDC
 1 x X520-2 Ethernet Server Adapter
VM configuration 3 x Oracle Linux VMs: The vCPU and memory

 VM version: 11 configuration was increased to 16
 OS: Oracle Linux 6.7 UEK3 vCPUs and 64GB RAM from
 16 vCPUs, 64GB RAM default values on CVM.
 CVM: 12 vCPUs, 32GB RAM
Oracle database Oracle 12c R1 RAC database: The SLOB tool was used to
 Oracle ASM (External Dependency) simulate real-world Oracle
 Oracle 12c EM Express database transactions.
 Silly Little Oracle Benchmark (SLOB)
Network 2 x Dell EMC Networking S4810 switches Dual redundant switches were
 Firmware: 9.10 used for high availability.
Performance  Nutanix Prism Performance monitoring for the

monitoring  VMware vCenter® performance monitoring XC Series cluster was done using
 Oracle 12c EM Express Nutanix Prism and vCenter.
Oracle database I/O was
monitored using Oracle 12c EM
Express which is shipped as part
of Oracle 12c.
C Technical support and resources
Dell.com/support is focused on meeting customer needs with proven services and support.
Dell TechCenter is an online technical community where IT professionals have access to numerous resources
for Dell EMC software, hardware and services.
Storage Solutions Technical Documents on Dell TechCenter provide expertise that helps to ensure customer
success on Dell EMC Storage platforms.
C.1 Related resources

See the following referenced or recommended resources related to this document:
 Oracle Best Practices Guide: http://go.nutanix.com/rs/nutanix/images/BP-

2000_Oracle_on_Nutanix_Best_Practices.pdf
 Nutanix Bible: http://nutanixbible.com/
 Nutanix blog: http://longwhiteclouds.com/
 Acropolis Block Services: http://go.nutanix.com/rs/031-GVQ-112/images/acropolis-block-services.pdf

Deploying Oracle12c Rac DB On Xcseries All Flash

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Deploying Oracle12c Rac DB On Xcseries All Flash

Hochgeladen von

Copyright:

Verfügbare Formate

Deploying Oracle 12c RAC Database on

Dell EMC XC Series All-Flash

A Dell EMC Best Practices Guide

January 2017 Initial release

Nutanix node architecture

1.2 XC Series all-flash

XC Nutanix Acropolis Block Services architecture

2.1 Physical system configuration

2.1.1 Oracle single instance database configuration

Oracle single-instance configuration

Oracle RAC configuration

Oracle RAC configuration using ABS

2.3 Network configuration

3.1.1 Processor and memory

3.1.2 XC Series storage container and VMware storage virtualization

Find more information on ABS at the Nutanix portal.

3.1.3 VM storage controller and virtual disks

Virtual adapter and vdisk configuration for Oracle database VM

Controller Adapter type Usage Virtual disk

Controller 0 LSI Logic Parallel Guest OS 1 x 100GB

Controller 1 Paravirtual Data files, redo logs 4 x 200GB

Controller 2 Paravirtual Archived logs 2 x 200GB

3.1.3.1 Shared access for virtual disks

3.1.3.3 Enabling virtual disk UUID

vSwitches and target use

vSwitchNutanix NA Intra CVM and svm-iscsi-pg Primary storage

vSwitch0 vmnic0 10GB Management network, Management traffic, VM

vSwitch1 vmnic2 10GB VM network: Oracle Private Oracle RAC

vSwitch2 vmnic4 10GB VM network: iSCSI Dedicated iSCSI traffic to

3.1.4.2 Management/inter-node network

3.1.4.3 Public VM network

3.1.4.5 iSCSI network

These attributes help simplify implementation for configurations such as LACP.

3.2 VM guest OS configuration for Oracle guidelines

1. Install the base OS from a virtual CD or ISO.

ACTION=="add", SUBSYSTEMS=="scsi", ATTRS{vendor}=="VMware ",

udevadm trigger --action=add

15. Verify the new setting:

yum list oracle-rdbms-servers*

b. Install the corresponding package version using yum install command.

3.3 Nutanix CVM

3.4.1 Oracle ASM for Oracle single instance or RAC

Some of the benefits of ASM include:

 Automatic file management

The following Oracle ASM best practices are recommended:

KERNEL=="sd*", PROGRAM="/lib/udev/scsi_id --page=0x83 --whitelisted --

To find out how to query the disk id/WWN, refer to appendix A.

ACTION=="add|change", KERNEL=="sd*", RUN+="/bin/sh -c '/bin/echo deadline

ACTION=="add|change", KERNEL=="sd*", RUN+="/bin/sh -c '/bin/echo noop >

Example ASM disk group configuration

DATA 4 1 MB Database files; temporary table space; online redo

FRA 2 1 MB Archive logs and backup data

CRS 1 1 MB Clusterware-related information such as the OCR

Example LVM configuration

DATA 4 /data Y Database files; temporary table space; online

FRA 2 /fra Y Archive logs and backup data

3.5 Performance monitoring

3.5.3 CVM CLI

1. ssh to the ESX host.

3.5.5 Oracle EM Express

 Database resource utilization

A.1 How to identify VMware virtual disks in Linux guest

[root@oracle-node10 ~]# ls -1 /dev/sd?|while read device; do echo -n