Oracle 12c RAC Administration Handout

1
Contents
Introduction to Oracle RAC ................................................................................................. 4
High Availability:................................................................................................................ 4
SMP & Cluster ..................................................................................................................... 5
Types of Clusters: ................................................................................................................ 5
Shared Storage in Clusters ................................................................................................. 6
Clusters in Market............................................................................................................... 7
Standalone vs RAC ............................................................................................................. 7
RAC Version History .......................................................................................................... 7
Storage Basics & ASM........................................................................................................... 8
DAS (Direct Attached Storage): ........................................................................................ 8
NAS ( Network Attached Storage): .................................................................................. 9
SAN(Storage Area Network):............................................................................................ 9
RAC Storage Options & File Systems: ........................................................................... 10
ASM..................................................................................................................................... 11
Oracle Real Application Cluster Basics ........................................................................... 13
Components of Oracle RAC ( Physical Layout ) .......................................................... 13
Components of Oracle RAC ( Software & Instance Layout ) ..................................... 14
Cache Fusion ...................................................................................................................... 15
Background Processes of Oracle 12c RAC..................................................................... 15
Oracle Clusterware (10gR2) ............................................................................................. 16
Oracle Grid Infrastructure (11gR2):................................................................................ 18
12c Oracle Real Application Clusters: ............................................................................ 21
Installing Oracle 12c RAC on OEL7 ................................................................................. 22
Installing Oracle 12CR1 RAC on OEL 7.2 Using Virtual Box at the LAB: ................ 22
Upgrading Database and Oracle RAC ............................................................................. 69
Necessity of Applying Patches........................................................................................ 88
How to Upgrade a Database: .......................................................................................... 89
Applying latest PSU to 12.1.0.2: ...................................................................................... 89
Applying Patches on RAC and Upgradations: ............................................................. 90
Applying 12.1.0.2 Patchset on 12.1.0.1: .......................................................................... 90
Clusterware Administration: ............................................................................................. 90

2
Workload Management ...................................................................................................... 94
Fast Connection Failover: ................................................................................................ 96
Services ............................................................................................................................... 96
Administering OCR & Voting Disk ................................................................................. 98
Administering Voting Disks ............................................................................................. 98
Recovering Voting Disks.................................................................................................. 98
Adding, Moving, or Deleting Voting Disks .................................................................. 98
Administering the Oracle Cluster Registry .................................................................... 99
Adding an Oracle Cluster Registry ................................................................................ 99
OCR Import/Export ....................................................................................................... 100
Configuring Initialization Parameters for an Oracle RAC Database ...................... 100
Troubleshooting Oracle RAC: ......................................................................................... 103
Debugging Resources: .................................................................................................... 103
Debugging Components: ............................................................................................... 103
Diagnostics Collection Script......................................................................................... 103
Oracle Clusterware Alerts ............................................................................................. 104
Troubleshooting OCR & VOTING DISK: .................................................................... 105
Handling Node Eviction Issues: ................................................................................... 105
OS Watcher: ..................................................................................................................... 106
12c RAC New Features: ..................................................................................................... 107
11gR2 vs 12c Database differences ................................................................................. 112
11gR2 vs 12c RAC differences ......................................................................................... 113
Flex Cluster & Flex ASM in 12c ....................................................................................... 113
Oracle 12c RAC Commands Changes ............................................................................ 120

3
 Introduction to Oracle RAC

Before understanding about Oracle RAC (Real Application clusters ), It's very important for
every DBA to know about Oracle High Availability.

High Availability:
If critical applications, servers, or data becomes unavailable, entire business might be
jeopardized. Loss of revenue, dissatisfied customers, penalties, and negative press will have
a lasting effect on organization's reputation. Hence High Availability plays a crucial role in
the modern IT Infrastructure. To understand about High Availability, we need to
understand about different types of failures. The Failures can be classified as shown below:

Downtime

Planned Unplanned

System Data Computer

Data Failure
Changes Changes Failure

Storage Human
Site Error Corruption
Failure Error

Each type of failure should be handled in a different way and Oracle have different
solutions for different types of failures. The overview of the high availability solutions is
given below:

 Oracle Real Application Clusters: For handling Computer Failures, Performance, &
Scalability
 Oracle ASM: Storage Failure ( Volume Manager )
 Oracle Data Guard: Site Failure ( Disaster Recovery )
 Oracle Flashback: Human Error
 HARD, RMAN & Secure Backup: Data Corruption

When we use all the High Availability (HA) solutions then we call it "Maximum Availability
Architecture (MAA)".

4
SMP & Cluster
Symmetric Multiprocessing (SMP) involves a multiprocessor computer hardware
architecture where two or more identical processors are connected to a single shared main
memory and are controlled by a single OS instance. A computer Cluster consists of a set of
loosely connected computers that work together so that in many respects they can be viewed
as a single system. Requires a software to manage the cluster.

Cluster has the following advantages over an SMP:

 Fault Tolerance.: High Availability: Avoids single point of failure.

 Cost-Effective
 Scale Up/ Scale Down

Types of Clusters:
Clusters can be classified into the following categories:

 Active-Active Cluster
 Active-Passive Cluster

Active-Active Cluster Active-Passive Cluster

1 All the cluster nodes access the storage Only 1 node is active always
simultaneously and communicate with
each other using Interconnect
2 If one of the node fails, the transactions are If the active node fails, the storage control is
failed over to the existing nodes, mostly given to the other node. This process is
transparently. called Failover. Failover occurs with the
help of Virtual IP.

3 Additional Nodes can be added to the No use of adding additional nodes as only
Cluster when required one node accesses the storage always

4 High Scalability No Scalability

5 Load Balancing and Better Performance No Load Balancing and Poor Performance
6 e.g. IBM HACMP e.g. Oracle RAC

5
Shared Storage in Clusters
Clusters can be classified into the following categories based on how the shared storage is
configured.

Shared Nothing:
 Data partitioned is accessed by several nodes separately.
 Each node can write into its own disks.
 Adding nodes dynamically is not possible. Requires downtime and sometimes re-
configuration
 No Failover capacity e.g. Teradata, Microsoft Cluster

Shared Everything:
 All nodes of the cluster access the same data simultaneously performing read/write
operations
 In case of node failure, the surviving nodes can recover the transactions. There is no
down-time.
 Nodes can be removed or added dynamically
 e.g. Oracle RAC, Oracle Exadata

6
Clusters in Market
Solaris : Sun Cluster, Veritas Cluster Services
HP-UX : HP Service Guard
HP Tru 64 : Tru Cluster
Windows : Microsoft Cluster Services
Linux: Oracle Clusterware
IBM AIX : HACMP

Standalone vs RAC

Key Points about Oracle RAC:

1. It is an option to Oracle Database for achieving clustering and for better performance
and high availability.
2. Oracle RAC allows multiple computers to run Oracle RDBMS software simultaneously
while accessing a single database, thus providing clustering.
3. All computers called as nodes in RAC see, access and update same set of data.
4. Each node has a separate database instance with its own Background Processes and
Memory Structures thus functioning like Standalone Database machine.
5. In-case of a node failure, the connections failover to one of the cluster nodes
transparently which is called as TAF thus providing High-Availability.
6. Nodes can be added or removed on the go easily thus providing Scale Up/ Down
feature.
7. Oracle Cluster Software/Grid Infrastructure will be installed on all the Nodes

RAC Version History

 Database clustering has been introduced in Oracle 6 and called as OPS ( Oracle Parallel
server) from 6.2
 Several new features have been added to OPS in 7,8 and 8i. Introduced Cache Fusion in
8i

7
 OPS has been re-named as RAC ( Real Application Clusters) from Oracle 9i. Cache-
Fusion has been improvised
 Oracle Cluster Manager has been introduced in 9i ( Supports only Windows and Linux),
prior to which there is no Oracle Proprietary Cluster Software. So Oracle has to depend
on other vendors.
 In 2003, with 10g, Oracle Clusterware has been introduced
 In 2005, with 10gR2 RAC, CVU has been introduced
 11gR1 also used Oracle Clusterware
 11gR2 introduced Oracle Grid Infrastructure which includes Oracle Clusterware as well
as ASM.
 Oracle RAC is certified on most platforms and till 100 Nodes

Storage Basics & ASM

Storage plays a key role in the Enterprise Infrastructure. Depending on the requirement of
the Project, different storage options can be considered. Storage plays a crucial role in
Database Projects.

A storage architecture defines the way Servers are connected to the storage array units.
Following are some the important storage architectures:

DAS (Direct Attached Storage):

Direct Attached Storage(DAS) is the most popular, common storage device which is directly
connected to the Host/Node. Some key points to remember:

 DAS should located physically close to the Node due to the limitations of Distance
and Performance of I/O bus

8
  The number of Storage Devices that can be connected is limited so it's not a scalable
option.

NAS ( Network Attached Storage):

NAS Devices are file servers which provide file I/O access where as SAN and DAS provide
block I/O storage access. Some key points to remember:

 NAS Devices are relatively inexpensive, easy to manage/deploy

 NAS Devices contain NIC Cards and are assigned with an IP address thus look like a
node in the network. The System in the network access the NAS devices through
standard LAN. So NAS Devices performance is impacted by the network bandwidth
 It's a scalable solution as more NAS devices can be added in the network when
required

SAN(Storage Area Network):

SAN is a dedicated network that provides access for Hosts/Machines to block level data
storage such as Disk Arrays, Tape Libraries and make them appear like locally attached
devices to the hosts.

 The SAN Storage is available to multiple hosts at the same time which makes SAN
Scalable and Flexible.
 SAN provides block level I/O access to the hosts where as NAS Provides File Level
I/O Access.
 The initial implementation cost is high and requires specialized training to
implement and manage.
 SAN provides High Performance, Reliability and High Availability.

9
  SAN I/O traffic does not impact the LAN Traffic and the Storage generally not
accessible/reachable by End Users/Applications.

RAC Storage Options & File Systems:

Different storage options that can be used for a RAC database are:

 RAW Devices
 CFS ( Cluster File System)
 ASM ( Automatic Storage Management)

Note: RAW devices are not supported anymore in 11gR2.

RAW Devices:
A disk or partition where all I/O operations to that device bypass any OS-level caches or
buffers and are delivered immediately to the driver subsystem

Examples: /dev/raw/raw1, /dev/rdsk/c0t0d0s0

 Raw devices are disk drives without any file system setup.
 Generally found in /dev in unix & linux bases OS’s and are supported for RAC as
they can be shared.
 A raw partition can have only 1 file stored in it which is a major disadvantage for
databases with several Datafiles or other files. Managing RAW devices is complex.
 RAW devices give better performance compared to other options as there will be no
additional overhead of filesystem.
 11gR2 doesn’t support RAW devices during installation but RAW devices can be
used during Upgrades if already being used in older versions

Cluster File Systems:

Several 3rd party vendors such as IBM, HP, Sun supplied Cluster file Systems are supported
by RAC.

Oracle Cluster File System (OCFS) presents a consistent file system image across the servers
in a cluster. OCFS allows administrators to take advantage of a files system for the Oracle
database files (data files, control files, and archive logs) and configuration files. This eases
administration of the Oracle Real Application Clusters.

OCFS is Oracle proprietary cluster filesystem which is certified only for Linux and Windows
OS’s. OCFS is used rarely in Real-Time implementations. OCFS is not very scalable for
large number of nodes. OCFS has 2 versions: OCFS and OCFS2.

ASM:
ASM (Automatic Storage Management) was introduced in Oracle 10g. ASM provides both
File System and Volume Management capabilities and simplifies the storage management
for DBAs. Most implemented solution in Real-Time from 10g.

10
NFS ( Network File System):
Started by Sun Microsystems as a generic fileserver solution. Originally UNIX-only, some
windows support available today. In the database world, generally dismissed as slow and
unreliable. NFS is certified in both 10g and 11g. It is a less expensive solution for RAC and
easy to use. In Real-Time NAS is not used in large-scale transaction-intensive applications.
NFS supports all types of files to be stored.

Depending on the requirements of the Project, a Valid Storage need to be used.

ASM
ASM (Automatic Storage Management) was introduced in Oracle 10g. ASM provides both
File System and Volume Management capabilities and simplifies the storage management
for DBAs.

ASM Instance:
An ASM Instance is built on the same technology as an Oracle Database Instance with its
own SGA and Background Processes. However, an Oracle Database Instance is responsible
for processing the user database requests and an ASM Instance mounts the Disk Groups it
manages and makes the ASM Files available to the Database Instances.

ASM Disks:
ASM Disks are storage devices like a Disk or a partition or a logical volume etc.

ASM Disk Group:

A Disk Group consists of multiple disks and is the fundamental object that ASM manages.

ASM Files:
These are the files that are stored in ASM Disk Groups. Each ASM file is contained within a
single ASM disk group. Some of the examples of ASM Files are Datafiles, Control Files,
SPFILE, RMAN Backup Files, Online Redo Logs, Archive logs etc

Example:
Have a look at the picture below:

11
In the image, there is an ASM Instance which is being used by 3 Databases. The files(Can a
Data File, Controlfile, Backups etc) related to the 3 databases will be stored in the 2 Disk
Groups A and B. Disk Group A has 3 Disks where as B has only 2 Disks.

ASM with RAC:

Most of the Oracle RAC Configurations use Oracle ASM. Each node in Oracle RAC should
have an ASM Instance. In the above Example, 2 RAC Databases(Each with 2 Nodes) and a
Standalone Database Node are accessing the ASM storage with 2 Disk Groups.

Advantages of ASM:
 Oracle proprietary solution addressing both File System and Volume Manager
requirements thereby reducing the dependency on various other File Systems and
3rd party volume managers.
 ASM Disk and Disk Group management ( Storage Management ) can be done by
even Oracle DBAs with minimum knowledge on storage using ASMCMD,
Enterprise Manager and even simple SQL Statements thereby reducing the
dependency on dedicated storage administrators.
 As a Volume Management solution, it provides Striping and Mirroring features for
better performance and redundancy respectively.
 Compatible and certified with Oracle Real Application Clusters (RAC) and RMAN.

12
 Oracle Real Application Cluster Basics

Components of Oracle RAC ( Physical Layout )

13
 All the nodes in the cluster will be connected to the Shared Storage to access the same
database.
 All nodes will be having atleast 2 NIC Cards: One for Public Network through which the
Clients connect and the other is Privtae Inter-Connect which is used to for Inter-Node or
Inter-Instance communication. For e.g. Cache Fusion
 Virtual IP, Introduced in 10g will be there for every node which helps in TAF
(Transparent Application Failover).

Components of Oracle RAC ( Software & Instance Layout )

 Each node in Oracle RAC has 2 Oracle Homes. 1: Oracle Clusterware Home 2: Oracle
RDBMS Home.
 Each node has 2 Instances. 1: Database Instance 2: ASM Instance. ASM is an Optional
Storage Option.
 Voting Disk and OCR are Oracle Clusterware related files which are in shared
storage and should be accessible by all the nodes in the cluster.
 The above picture shows the order of installation and configuration as well which
implies that once the Nodes have OS installed, Oracle Clusterware is installed 1st
followed by Oracle Database Software, ASM and then Oracle Database Creation.

14
Cache Fusion
Cache Fusion involves the ability of Oracle RAC to "fuse" i.e. Combine the in-memory data
cached physically separately on each computer into a single, Global Cache. Cache Fusion
happens with the help of the Private Interconnect.

Cache Fusion improves the performance of the databases as the I/Os are very less when
compared to the previous versions.

Background Processes of Oracle 12c RAC

The GCS and GES processes, and the GRD collaborate to enable Cache Fusion. The Oracle
RAC processes and their identifiers are as follows:

 ACMS: Atomic Controlfile to Memory Service (ACMS)

In an Oracle RAC environment, the ACMS per-instance process is an agent that contributes
to ensuring a distributed SGA memory update is either globally committed on success or
globally aborted if a failure occurs.

 GTX0-j: Global Transaction Process

The GTX0-j process provides transparent support for XA global transactions in an Oracle
RAC environment. The database autotunes the number of these processes based on the
workload of XA global transactions.

 LMON: Global Enqueue Service Monitor

The LMON process monitors global enqueues and resources across the cluster and performs
global enqueue recovery operations.

 LMD: Global Enqueue Service Daemon

The LMD process manages incoming remote resource requests within each instance.

 LMS: Global Cache Service Process

The LMS process maintains records of the data file statuses and each cached block by
recording information in a Global Resource Directory (GRD). The LMS process also controls
the flow of messages to remote instances and manages global data block access and
transmits block images between the buffer caches of different instances. This processing is
part of the Cache Fusion feature.

 LCK0: Instance Enqueue Process

The LCK0 process manages non-Cache Fusion resource requests such as library and row
cache requests.

15
  RMSn: Oracle RAC Management Processes (RMSn)

The RMSn processes perform manageability tasks for Oracle RAC. Tasks accomplished by
an RMSn process include creation of resources related to Oracle RAC when new instances
are added to the clusters.

 RSMN: Remote Slave Monitor manages background slave process creation and
communication on remote instances. These background slave processes perform
tasks on behalf of a coordinating process running in another instance.

Oracle Clusterware (10gR2)

Oracle Clusterware is a software installed on a group of nodes running in the same physical
cluster.

The Files that belong to Oracle Clusterware and that should be in shared storage are OCR
and Voting Disk.

The Processes that belong to Oracle Clusterware are:

• Cluster Synchronization Services Daemon - OCSSd

• Cluster Ready Services Daemon - CRSd
• Event Manager Daemon – EVMd
• Process Monitor Daemon – OPROCd (Obsolete in 11.2 and called as cssdagent)

Oracle Clusterware is installed, entries are added to the /etc/inittab file , the init process
spawns the EVMD, OCSSD, and CRSD processes.

OCR:
Oracle Clusterware uses the Oracle Cluster Registry (OCR) to store and manage information
about the components that Oracle Clusterware controls, such as Oracle RAC databases,
Instances, Listeners, VIPs, and services and any applications.

To ensure cluster high availability, Oracle recommends that you define multiple OCR
locations (multiplex).

1. Oracle Clusterware reads the ocr.loc file (located in the /etc directory on Linux and Unix
systems to determine which applications resources need to be started and the nodes on
which to start them.
2. Oracle uses a distributed shared cache architecture to optimize queries against the
cluster repository. Each node maintains a copy of the OCR in memory.
3. You can have up to five OCR locations. Each OCR location must reside on shared
storage that is accessible by all of the nodes in the cluster
4. You can replace a failed OCR location online if it is not the only OCR location
5. The OCR clients are Oracle Enterprise Manager, the Server Control Utility (SRVCTL), the
OCR configuration utility (OCRCONFIG), OUI or the Database Configuration Assistant
(DBCA)

16
Voting Disks:
Manages node membership by using a health check and arbitrates cluster ownership among
the instances in case of network failures. RAC uses the voting disk to determine which
instances are members of a cluster. The voting disk must reside on shared disk.

If you configure voting disks on Oracle ASM, then you do not need to manually configure
the voting disks. Depending on the redundancy of your disk group, an appropriate number
of voting disks are created.

You should have at least three voting disks, unless you have a storage device, such as a disk
array that provides external redundancy. Oracle recommends that you do not use more than
five voting disks. The maximum number of voting disks that is supported is 15.

CRSd
 CRS is installed and run from a different ORACLE_HOME known as ORA_CRS_HOME
( Called as GRID_HOME starting from 11gR2), which is independent of
ORACLE_HOME.
 CRSd manages the resources like starting and stopping the services and failing-over the
cluster resources which can be Virtual IP, Database Instance, Listener or Database etc
 CRS daemon has two modes of running. During startup and after a shutdown. During
planned clusterware start it is started as ‘reboot’ mode. It is started as ‘restart’ mode after
unplanned shutdown in which it retains the previous state and returns resources to their
previous states before shutdown.
 In case of failure of CRSd, it starts automatically without restarting the node.
 Manages the Oracle Cluster Registry and stores the current known state in the Oracle
Cluster Registry.
 Runs as ‘root’ on Unix and ‘LocalSystem’ on windows and automatically restarts in case
of failure.
 CRS requires the public interface, private interface and the Virtual IP (VIP) for the
operation. All these interfaces should be up and running, should be able to ping each
other before starting CRS Installation. Without the above network infrastructure CRS
cannot be installed.
 Command: ps -ef | grep crsd

OCSSd
It maintains membership in the cluster through a special file called a voting disk (also
referred to as a quorum disk). This is the first process that is started in the Oracle
Clusterware stack.

 OCSS in Stand-alone Databases using ASM is used for the inter-instance communication
and in RAC environments, identified a Clustered Configuration.
 OCSS reads OCR to locate VD and reads the VD to determine the number and names of
cluster members.
 CSS verifies the number of nodes already registered as part of the cluster. After
verification, if no MASTER node has been established, CSS authorizes the verifying node
to be the MASTER node. This is the first node that attains the ACTIVE state. Cluster
synchronization begins when the MASTER node synchronizes with the other nodes
17
 OCSSd offers Node Membership(NM) and Group Membership(GM) services.
 The NM checks the heartbeat across the various nodes in the cluster every second. If the
heartbeat/node members do not respond within 60 seconds, the node (among the
surviving nodes) that was started first (master) will start evicting the other node(s) in the
cluster.
 All clients that perform I/O operations register with the GM (e.g., the LMON, DBWR).
Reconfiguration of instances (when an instance joins or leaves the cluster) happens
through the GM. When a node fails, the GM sends out messages to other instances
regarding the status.

EVMd
It receives the FAN events posted by the clients and propagates the information to the other
nodes. Failure of EVMd doesn't require node reboot and restarts automatically.

It is spawned by init.evmd wrapper script. It starts evmlogger child process which scans the
callout directory and starts racgevt process to execute the callouts.

ONS:

It is a publish and subscribe service for communicating Fast Application Notification (FAN)
events to clients. Whenever the state of resource changes in the cluster nodes, CRS triggers
a HA event and routes them to the ONS process which propagates the information to other
cluster nodes

OProcd
OPROCd serves as the I/O fencing solution for the Oracle Clusterware. It is the process
monitor for Oracle Clusterware and it uses the hang check timer for the cluster integrity so
that the hanging nodes cannot perform any I/O. Failure of the OPROCd process causes the
node to restart.

Oracle Grid Infrastructure (11gR2):

Several new Background processes and components have been added to 11gR2 because of
several new features.

 Oracle Clusterware is started by OS Init Daemon

 The following picture shows the startup sequence in 11gR2:

18
 In 11gR2 Clusterware Processes have systematically organized into several
component groups as show below:

 When you have Oracle RAC installed, the crsd process monitors the Oracle
database instance, listener, and so on, and automatically restarts these
components when a failure occurs. When a CRS resource fails, the CRS daemon
attempts to restart it, if the resource is so configured. CRS fails the resource over
to another node (again, if it is configured to do so) after exhausting restart
attempts.
 Cluster Synchronization Service (CSS): Manages the cluster configuration by
controlling which nodes are members of the cluster and by notifying members
when a node joins or leaves the cluster. If you are using certified third-party
clusterware, then CSS processes interfaces with your clusterware to manage node
membership information. CSS has three separate processes: the CSS daemon
(ocssd), the CSS Agent (cssdagent), and the CSS Monitor (cssdmonitor). The
cssdagent process monitors the cluster and provides input/output fencing. This
service formerly was provided by Oracle Process Monitor daemon (oprocd), also
known as OraFenceService on Windows. A cssdagent failure results in Oracle
Clusterware restarting the node.
 Disk Monitor daemon (diskmon): Monitors and performs input/output fencing
for Oracle Exadata Storage Server. As Exadata storage can be added to any
Oracle RAC node at any point in time, the diskmon daemon is always started
when ocssd is started.
 Multicast domain name service (mDNS): Allows DNS requests. The mDNS
process is a background process on Linux and UNIX, and a service on Windows.
 Oracle Grid Naming Service (GNS): Is a gateway between the cluster mDNS
and external DNS servers. The GNS process performs name resolution within the
cluster.
 oraagent: Extends clusterware to support Oracle-specific requirements and
complex resources. It runs server callout scripts when FAN events occur. This
process was known as RACG in Oracle Clusterware 11g Release 1 (11.1).

19
 Oracle root agent (orarootagent): Is a specialized oraagent process that helps
CRSD manage resources owned by root, such as the network, and the Grid
virtual IP address
 Cluster kill daemon (oclskd): Handles instance/node evictions requests that
have been escalated to CSS
 Grid IPC daemon (gipcd): Is a helper daemon for the communications
infrastructure

The overall Clusterware Stack startup Sequence can be understand using the below
picture:

20
12c Oracle Real Application Clusters:

 Oracle Flex ASM - This feature of Oracle Clusterware 12c claims to reduce per-node
overhead of using ASM instance. Now the instances can use remote node ASM for any
planned/unplanned downtime. ASM metadata requests can be converted by non-local
instance of ASM.

 ASM Disk Scrubbing - From RAC 12c, ASM comes with disk scrubbing feature so that
logical corruptions can be discovered. Also Oracle 12c ASM can automatically correct
this in normal or high redundancy diskgroups.
 Oracle ASM Disk Resync & Rebalance enhancements.
 Application Continuity (AC) - is transparent to the application and in-case the database
or the infrastructure is unavailable, this new features which work on JDBC drivers,
masks recoverable outages. This recovers database session beneath the application so
that the outage actually appears to be delayed connectivity or execution. Transaction
guard (improvements of Fast Application Notification).
 IPv6 Support - Oracle RAC 12c now supports IPv6 for Client connectivity, Interconnect
is still on IPv4.
 Per Subnet multiple SCAN - RAC 12c, per-Subnet multiple SCAN can be configured per
cluster.
 Each RAC instance opens the Container Database (CDB) as a whole so that versions
would be same for CDB as well as for all of the Pluggable Databases (PDBs). PDBs are
also fully compatible with RAC.
 Oracle installer will run root.sh script across nodes. We don't have to run the scripts
manually on all RAC nodes.
 new "ghctl" command for patching.
 TAF is extended to DML transactions failover (insert/update/delete).

21
 Installing Oracle 12c RAC on OEL7

Installing Oracle 12CR1 RAC on OEL 7.2 Using Virtual Box at the LAB:

The following explains how to install a 2 nodes Oracle 12c Real Application Cluster
(RAC) on Oracle Linux 7. I did this installation on Oracle Virtual Box by creating 2
virtual machines with shared storage. OS platform I used is Oracle Enterprise Linux
7, and Oracle GI and RDBMS version is 12.1.0.2.

My environment:

Rac1 - Oracle Enterprise Linux 7 (64 bit)

Rac2 - Oracle Enterprise Linux 7 (64bit)
Dns - Oracle Enterprise Linux 4 (32bit)

Prerequisites

Prepare all the nodes by installing Oracle Linux 7 with all selected packages during
installation. Have private interconnects setup, and shared storage mounted on all the
nodes. Have 2 public IPs, 2 virtual IPs, 2 private IPs, 1 dns IP and 3 SCAN IP (from
public subnet), which we will use later during the installation. For this article, node
names for are as follows

Public
racnode1.oracle.com
racnode2.oracle.com

Virtual
racnode1-vip.oracle.com
racnode2-vip.oracle.com

Private
racnode1-priv.oracle.com
racnode2-priv.oracle.com

Dns
dns.oracle.com

Recommended way for SCAN configuration is to have 3 IP addresses which are

resolved through DNS.

Download and unzip the software

Download 12c Grid Infrastructure and RDBMS software’s. First 2 zip files are for
database and last 2 are for Grid Infrastructure binaries. For this article, I extracted
the software under /u01/stage/ directory.

22
Perform Oracle RAC Installation Prerequisites:

Now we start the installation. Following are the steps to install Oracle 12c RAC on Linux.
Here we are assuming that all the nodes are installed with all packages and up and running.

1) On each node, edit /etc/selinux/config and set value for SELINUX to either
“permissive” or “disabled”

SELINUX=disabled

2) On each node, configure Shared Memory File System. Add following line in /etc/fstab
file for shared memory file system. Modify the value of “size” based on the amount of
memory you will be using for your SGA.

tmpfs /dev/shm tmpfs rw,exec,size=4g 00

3) On each node, disable the firewall.

[root@racnode1 ~]# systemctl start firewalld.service

[root@racnode1 ~]# systemctl stop firewalld.service
[root@racnode1 ~]# systemctl disable firewalld.service
rm '/etc/systemd/system/basic.target.wants/firewalld.service'
rm '/etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service'

4) Reboot all the nodes

5) On each node, in /etc/host file, add the IP addresses and fully qualified names of each
node of RAC, including public IPs; virtual IP; private IPs .

#Public
192.168.1.101 racnode1.oracle.com racnode1
192.168.1.102 racnode2.oracle.com racnode2
#Virtual
192.168.1.201 racnode1-vip.oracle.com racnode1-vip
192.168.1.202. racnode2-vip.oracle.com racnode2-vip

#Private
192.168.2.101 racnode1-priv.oracle.com racnode1-priv
192.168.1.102 racnode2-priv.oracle.com racnode2-priv

#Dns
192.168.1.100 dns.oracle.com dns

Make sure that Public interface and Private interfaces have the same name and they list in
same order on all the nodes, in ifconfig command (my public interface name is enp0s3 and
private interface name is enp0s8)

23
6) Test the nodes connectivity with each other using ping command from each node to the
other node(s). SCAN IP and Virtual IPs are not required to be tested at this point.

From node racnode1

[root@racnode1 ~]# ping racnode1
[root@racnode1 ~]# ping racnode1-priv
[root@racnode1 ~]# ping racnode2
[root@racnode1 ~]# ping racnode2-priv
[root@racnode1 ~]# ping dns

From node racnode2

[root@racnode2 ~]# ping racnode2
[root@racnode2 ~]# ping racnode2-priv
[root@racnode2 ~]# ping racnode1
[root@racnode2 ~]# ping racnode1-priv
[root@racnode2 ~]# ping dns

7) We can perform automatic configuration of the node using “yum” command. If you want
to do manual configuration, skip this step and go to next step.

On each node, execute following command to perform all prerequisites automatically.

[root@racnode1 ~]# yum install oracle-rdbms-server-12cR1-preinstall –y

As already mentioned, above command will install all required packages which are needed
for grid infrastructure and/or RDBMS software installation.

Create the new groups and users.

groupadd -g 54321 oinstall

groupadd -g 54322 dba
groupadd -g 54323 oper

useradd -u 54321 -g oinstall -G dba,oper oracle

Add the following lines to the "/etc/security/limits.conf" file.

# Oracle user
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft nproc 16384
oracle hard nproc 16384
oracle soft stack 10240
oracle hard stack 32768
oracle hard memlock 134217728
oracle soft memlock 134217728

24
8) Skip this step if you have followed Step 7 above; otherwise perform following tasks
on each node.
Install following RPM packages (or latest version) from either yum repository or from Linux
7 media

binutils-2.23.52.0.1-12.el7.x86_64
compat-libcap1-1.10-3.el7.x86_64
gcc-4.8.2-3.el7.x86_64
gcc-c++-4.8.2-3.el7.x86_64
glibc-2.17-36.el7.i686
glibc-2.17-36.el7.x86_64
glibc-devel-2.17-36.el7.i686
glibc-devel-2.17-36.el7.x86_64
libaio-0.3.109-9.el7.i686
libaio-0.3.109-9.el7.x86_64
libaio-devel-0.3.109-9.el7.i686
libaio-devel-0.3.109-9.el7.x86_64
ksh
make-3.82-19.el7.x86_64
libXi-1.7.2-1.el7.i686
libXi-1.7.2-1.el7.x86_64
libXtst-1.2.2-1.el7.i686
libXtst-1.2.2-1.el7.x86_64
libgcc-4.8.2-3.el7.i686
libgcc-4.8.2-3.el7.x86_64
libstdc++-4.8.2-3.el7.i686
libstdc++-4.8.2-3.el7.x86_64
libstdc++-devel-4.8.2-3.el7.i686
libstdc++-devel-4.8.2-3.el7.x86_64
sysstat-10.1.5-1.el7.x86_64
unixODBC-2.3.1

On each node, edit /etc/sysctl.conf add following entries to set kernel parameters

fs.file-max = 6815744
kernel.sem = 250 32000 100 128
kernel.shmmni = 4096
kernel.shmall = 1073741824
kernel.shmmax = 4398046511104
kernel.panic_on_oops = 1
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048576
fs.aio-max-nr = 1048576
net.ipv4.ip_local_port_range = 9000 65500

25
Execute following command after adding above lines

/sbin/sysctl -p

On each nodes, edit the /etc/security/limits.d file to set shell limits for oracle user.

# Oracle user
oracle soft nofile 1024
oracle hard nofile 65536
oracle soft nproc 16384
oracle hard nproc 16384
oracle soft stack 10240
oracle hard stack 32768
oracle hard memlock 134217728
oracle soft memlock 134217728

9) On each node, install oracleasm-support and oracleasmlib rpm’s , and then configure
oracleasm.
oracleasm kernel driver is built in Oracle Linux and does not need to be installed.
Install oracleasm-support from yum repository or from Linux media, then download
oracleasmlib package form the following URL, and install.
http://www.oracle.com/technetwork/server-storage/linux/asmlib/ol7-2352094.html

[root@racnode1 ~]# yum install oracleasm-support

[root@racnode1 ~]# rpm -i oracleasm-lib*

26
10) On each node, stop and disable NTP (Network Time Protocol)
During the installation, Cluster Time Synchronization Service would be installed and used
instead or NTP.

[root@racnode1 ~]# /sbin/service ntpd stop

[root@racnode1 ~]# chkconfig ntpd off
[root@racnode1 ~]# mv /etc/ntp.conf /etc/ntp.conf.org
[root@racnode1 ~]# rm /var/run/ntpd.pid

11) DNS configuration:

Add the /etc/hosts file as shown below.

Add the scan name, scan IP’s and all the racnode IP’s in the /var/named/localdomain.zone
file as follow. Here I used scan name as ‘rac-scan’.

27
Enable and restart the named service on the dns server a follow.

[root@dns ~] chkconfig named on

[root@dns ~] service named start
[root@dns ~] service named restart

12) On each rac node, make sure /etc/resolv.conf file contain entries similar to the
following. Replace oracle.com with your domain name and 192.168.1.100 with your names
server IP address

search localdomain
nameserver 192.168.1.100

Execute the following command as root user to prevent overwriting the /etcresolv.conf on
network reboot or system reboot.

chattr +i /etc/resolv.conf

With this in place and the DNS configured the SCAN address is being resolved to all three
IP addresses on each node.

28
29
13) On each rac node, install cvuqdisk RPM
This is required for cluvfy (Cluster Verification Utility) to work properly to discover shared
disks, otherwise an error will be reported while running cluvfy. Log in as root and execute
following steps

Set the environment variable CVUQDISK_GRP to point to the group what will own
cvuqdisk, which is oinstall group
[root@racnode1 ~]# export CVUQDISK_GRP=oinstall

Install cvuqdisk which will be found under unziped Grid Infrastructure software

[root@racnode1 ~]# cd /u01/stage/grid/rpm

[root@racnode1 ~]# rpm –ihv cvuqdisk-1.0.9-1.rpm
[root@racnode1 ~]# scp cvuqdisk-1.0.9-1.rpm racnode2:/u01/.

[root@racnode2 ~]# cd /u01/

[root@racnode2 ~]# rpm -ihv cvuqdisk-1.0.9-1.rpm

14) On each node, create directories and change ownership to respective users.

[root@racnode1 ~]# mkdir -p /u01/app/grid/12.1.0.2

[root@racnode1 ~]# mkdir -p /u01/app/oracle/product/12.1.0.2
[root@racnode1 ~]#

[root@racnode1 ~]# chown -R oracle:oinstall /u01/app/grid/12.1.0.2

[root@racnode1 ~]# chown oracle:dba /u01/app/oracle/product/12.1.0.2
[root@racnode1 ~]# chmod -R 775 /u01/

Shutdown all the racnodes.

15) From any racnode, create the shared disks as follows. You can use other ways also for
configure shared storage for all nodes.

30
 Go to setting and click on the storage and click button add hard disks to create a disk.

31
32
 Make the created disks as shared disks between all the node as follows.
From Virtual media manager settings, Select the disk and click on Modify button at the top
of screen.

33
 Select the Sharable and click on ok button.

 Go to racnode2 setting and select storage options and click on add hard disks button.
Click on choose existing hard disk.

34
 Select created sharable disks from the path.

35
 Start the machines and execute the below command as root user.

# fdisk -l

For simplicity, after executing command fdisk for each disk, press following sequence keys
n, p, 1, <enter>, <enter>, w to create new partitions for each disk

36
 After the partitioning completes, following are my partitioned disks

[root@racnode1 ~]# ls -ltr /dev/sd*

brw-rw---- 1 root disk 8, 0 Oct 16 17:48 /dev/sda
brw-rw---- 1 root disk 8, 1 Oct 16 17:48 /dev/sda1
brw-rw---- 1 root disk 8, 3 Oct 16 17:48 /dev/sda2
brw-rw---- 1 root disk 8, 2 Oct 16 17:48 /dev/sda3
brw-rw---- 1 root disk 8, 16 Oct 16 17:59 /dev/sdb
brw-rw---- 1 root disk 8, 32 Oct 16 17:59 /dev/sdc
brw-rw---- 1 root disk 8, 17 Oct 16 18:55 /dev/sdb1
brw-rw---- 1 root disk 8, 33 Oct 16 18:55 /dev/sdc1
[root@racnode1 ~]#

 Configure oracleasm in all the racnodes as below.

16) From any node, Create the disks as asmdisks as shown below by using oraclasm.

37
 Execute “oracleasm scandisks” and “oracleasm listdisks” command on all other nodes
also and you should be able to see all the ASM disks.
From the other node(s), issue following as root user

[root@racnode2 ~]# /etc/init.d/oracleasm start

[root@racnode2 ~]# /etc/init.d/oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...

[root@racnode2 ~]# /etc/init.d/oracleasm listdisks

ASMDATA1
ASMDATA2

38
17) On each node, add IO scheduler for ASM disks. Modify highlighted disks names to
match your environment.

[root@racnode1 ~]# echo deadline > /sys/block/sdb/queue/scheduler

[root@racnode1 ~]# echo deadline > /sys/block/sdc/queue/scheduler

18) Configure your UDEV rules, as shown here in racnode1.

 Add the following to the "/etc/scsi_id.config" file to configure SCSI devices as trusted.
Create the file if it doesn't already exist.

options=-g

 The SCSI ID of my disks are displayed below.

[root@racnode1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdb

1ATA_VBOX_HARDDISK_VBf72914ad-884ce4f5
[root@racnode1 ~]# /usr/lib/udev/scsi_id -g -u -d /dev/sdc
1ATA_VBOX_HARDDISK_VBeafc0594-4e0ec784

 Using these above values, edit the "/etc/udev/rules.d/99-oracle-asmdevices.rules" file

by adding the following 2 entries in below commands. All parameters for a single entry
must be on the same line.

39
[root@racnode1 ~]# KERNEL=="sd?1", SUBSYSTEM=="block",
PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",
RESULT=="1ATA_VBOX_HARDDISK_VBf72914ad-884ce4f5",
SYMLINK+="oracleasm/ASMDATA1", OWNER="oracle", GROUP="dba", MODE="0660"

[root@racnode1 ~]# KERNEL=="sd?1", SUBSYSTEM=="block",

PROGRAM=="/usr/lib/udev/scsi_id -g -u -d /dev/$parent",
RESULT=="1ATA_VBOX_HARDDISK_VBeafc0594-4e0ec784",
SYMLINK+="oracleasm/ASMDATA2", OWNER="oracle", GROUP="dba", MODE="0660"

 Load updated block device partition tables.

[root@racnode1 ~]# /sbin/partprobe /dev/sdb1

[root@racnode1 ~]# /sbin/partprobe /dev/sdc1

 Test the rules are working as expected.

# /sbin/udevadm test /block/sdb/sdb1

 Reload the UDEV rules.

# /sbin/udevadm control --reload-rules

19) On each node, add following in /home/<username>/.bash_profile file of both users;

“grid” and “oracle”. At this point, we do no need to set other environment variables i.e.
ORACLE_BASE or ORACLE_HOME.

umask 022

20) Run cluvfy (Optional)

In the node where you plan to initiate the installation, open a terminal and log in as “grid”
user, and you can optionally execute cluster verification utility (cluvfy).

[grid@racnode1 ~]$ cd /u01/stage/grid/

[grid@racnode1 ~]$ ./runcluvfy.sh stage -pre crsinst -n racnode1 -fixup -verbose

This will fail if user equivalence (passwordless ssh login) has not been configured from each
node to other node. Passwordless login using ssh is also optional at this point, because
installer will allow us to configure user equivalence during the installation wizard as you
will see in a while.

21) Start installation from the extracted software directory (/u01/stage/grid), execute
“./runInstaller” to begin the installation

40
 Select “Install and configure Oracle Grid Infrastructure for a cluster “ from the given
options and click on next button.

41
 Select “Configure a Standard cluster “and click on next.

 Select “Advance installation “and click next.

42
 Select a language from the available options.

 Enter the SCAN name and uncheck the configure GNS button.

43
 In this window click on “Add button” and add the second rac node public hostname
and virtual hostnames.

 Click on “SSH connectivity” and enter the password oracle user. Click on setup
button to configure ssh connectivity between the node. After successful ssh setup
click next button.

44
 Select the private and public interfaces for grid infrastructure and click on next
button.

 Accept the default “Use standard ASM storage” as we are going to configure grid
with ASM storage and click next.

45
 Select the storage and select the redundancy and show in below picture.

 Accept the default and click on next.

46
 If You want to configure the enterprise manager cloud control then select the “
Register with Enterprise manager (EM) Cloud Control “ or uncheck it and click next
to proceed installation.

47
 Enter the related groups for each in this section and go with next.

 Enter the Oracle base and Grid Software installation locations.

48
 Select the Inventory location and click on next button.

 If You want to run Root scripts automatically enter the password for root user or you
can manually run the root scripts.

49
 The window will perform the pre requisite checks and list the warnings and errors as
. Make sure that no errors will be present in this screen before doing to proceed with
installation. The below warnings can be ignored safely.

 Review the listed information and click on install button at the bottom.

50
  The installation will process and wait for the installation to complete.

 The successful installation will display the below screen. Click on close button to
close the installation window.

The 12.1.0.2 grid Installation completed successfully.

51
 Install the Database Software
Start the virtual machines racnode1 and racnode2. Check that all services are up using "crsctl
stat res -t", as described before.

 Create the required directories on both nodes and set the permissions for oracle user
as below.

 Login to racnode1 as the oracle user and start the Oracle installer from the database
software stage location.

$ cd /u01/stage/database

$ ./runInstaller

52
 Uncheck the security updates checkbox and click the "Next" button and "Yes" on the
subsequent warning dialog.

53
 Select the "Install database software only" option, then click the Next button.

 Accept the default "Oracle Real Application Clusters database installation" option by
clicking the "Next" button.

54
 Select the racnode2 and click on next.

 Select the languages from given list and click on next button.

55
 Select the "Enterprise Edition" option, then click the "Next" button.

 Give the locations for Oracle base and Oracle home installation location.

56
 Select the desired operating system groups, then click the Next button.

 It will perform prerequisite check and list the errors and warning. If there are no
warning it displays the summary information.

57
 Click on install button to start the installation.

 Wait for the installation to complete.

58
 When prompted, run the configuration script on each node as in give order. When
the scripts have been run on each node, click the "OK" button.

 The successful installation shows the below screen. Click on close button to exit.

59
 12c RAC Database creation using DBCA
Start the racnode1 and racnode2 machine and login as oracle user. Go the oracle home
software location and run “database configuration assistance”.

[ oracle@racnode1 ~] $ cd /u01/app/oracle/product/12.1.0.2/bin

[ oracle@racnode1 ~] $ ./dbca

 The database creation window will display as shown below. Select “create database”
and click on next button.

 Select “Advanced mode” in this screen and click on next.

60
 Select “General purpose or Transaction processing” and click in next .

61
 Enter the Global database name and SID for database. Click on next.

 Uncheck the box for “Configure Enterprise manager” and go with next to proceed.

62
 Enter the database administrative passwords and click on next.

 Accept the defaults by clicking on the next button.

63
 Select “Database files storage type” as ASM and make the required the changes.
Click on next button to continue installation.

 Change the database options as per your requirement and click on next button.

64
 Change the size of memory, character set and connection mode setting in this screen
as required and click on next button.

 Select “Create the database” and click on Next to proceed with database creation.

65
 The installation wizard will perform prerequisite checks and display the summary
screen as shown below. Click on “Finish” button to start the database creation.

 The database creation will process as below figure.

66
  When the database creation completed, click the "Close" button to exit installer.

 Check the RAC database installation status.

[root@racnode1 ~]# su - oracle

[oracle@racnode1 ~]$ export ORACLE_UNQNAME=+ASM

[oracle@racnode1 ~]$ export ORACLE_SID=+ASM1

[oracle@racnode1 ~]$ export ORACLE_BASE=/u01/app/oracle

[oracle@racnode1 ~]$ export ORACLE_HOSTNAME=racnode1.oracle.com

[oracle@racnode1 ~]$ export ORACLE_HOME=/u01/app/grid/product/12.1.0.2

[oracle@racnode1 ~]$ srvctl config database -d PROD

Database unique name: PROD

Database name: PROD

Oracle home: /u01/app/oracle/product/12.1.0.2

Oracle user: oracle

67
Spfile: +DATA/PROD/spfilePROD.ora

Password file: +DATA/PROD/orapwprod

Domain:

Start options: open

Stop options: immediate

Database role: PRIMARY

Management policy: AUTOMATIC

Server pools: PROD

Disk Groups: DATA

Mount point paths:

Services:

Type: RAC

Start concurrency:

Stop concurrency:

OSDBA group: dba

OSOPER group:

Database instances: PROD1, PROD2

Configured nodes: racnode1, racnode2

Database is administrator managed

[oracle@rac1 ~]$ srvctl status listener

Listener LISTENER is enabled

Listener LISTENER is running on node(s): racnode1,racnode2

[oracle@rac1 ~]$ srvctl status asm

ASM is running on racnode1,racnode2

68
 Upgrading Database and Oracle RAC
Necessity of Applying Patches
 A Database Internal Error/Bug is causing Application/Database unavailability
or Severe Performance Issue
 End of Support for a Particular Database version. For e.g 11gR2 Support
extended support ends in Jul-20. So the customers should consider upgrading to
12c seriously. Customers can apply Patchsets or Upgrading to any new Major
Release based on the requirement
 A CPU/PSU patch needs to be applied
 Due to EOL for Hardware or for reduction overall TCO, Customers might
consider H/W as well as Platform migration which requires to Install Softwares ,
Applying Patches, Patchsets etc
 For getting new features which Improves Performance, Stability, Administration.
For e.g. ACFS in 11g, AWR & ASH in 10g etc

Types of Patches:

Type of Patches Description

One-Off Patch This is a single patch to fix a specific problem.

Merge level request patch In case of Conflict with existing one-off, Oracle
(MLR) development can be asked to provide a merged set of
these patches.

Critical Patch Updates (CPU) Security / Performance / Highly recommended Patches

released 4 Times a Year

Patch Set Updates (PSU) Patch Set Updates bundle previously separate patches in
given product areas together

e.g. 11.2.0.4.1 Patch Set Update (PSU)

Patch set To apply a Point Release. For e.g. Patch set for Upgrading
from 11.2.0.3 to 11.2.0.4

Major Release Update To upgrade to the next/latest major release which

requires higher Downtime/Testing than all other Patch
Types.

69
How to Upgrade a Database:
There are different methods to upgrade an Oracle Database but the DBA has to
choose the Best method based on the Current Database Size, Version , OS, Data
Complexity, Downtime etc

DBUA
Manual Upgrade
Data Pump Export & Import , Export & Import
TTS
Oracle Data Guard SQL Apply
Oracle Streams & Golden Gate

DBUA is the most preferred method. Here we are going to demonstrate how we can
upgrade a real application cluster database to 12.1.0.2 using DBUA.

My environment: 2 node RAC running on 11.2.0.4 clusterware and database

Node1: racn1.dbatalent.com - Oracle Enterprise Linux 6 (64 bit)
Node2: racn2.dbatalent.com - Oracle Enterprise Linux 6 (64bit)

Section 1: Grid Infrastructure (clusterware) upgradation to 12c

 Initially verifying the clusterware version before upgradation.

 Create required directories for holding new 12.1.0.2 clusterware installation

and assign groups and permissions as shown below,

 Copy and unzip Oracle grid and database 12.1.0.2 software and run Oracle
universal installer as oracle user

70
 Select the option to upgrade the clusterware as below,

 Select the desired language and in the next screen setup ssh connectivity
between the nodes and proceed to the next screen

71
 If your environment has Oracle cloud control configured and if you wish to
add the clusterware as a managed target, enter the required configuration
details otherwise skip this step and proceed. The clusterware can be added as
managed target even after the upgradation completes.

72
 Select the OS groups that will be used for OS authentication to ASM

 Select the location of Oracle base and give the path for new clusterware
location which was created earlier.

73
 From 12c, OUI is capable of running post install root scripts if we supply root
credentials in this step. Or you can choose to manually run root scripts by
clicking next.

74
 In 12c, root scripts can be executed in batches. Root scripts are executed
simultaneously on every node in a batch and batches are run one after the
other and will ask for confirmation before executing the scripts in a new
batch. Batches can be specified based on number of nodes and you can specify
up to 3 batches.

 Oracle universal installer will now perform prerequisite checks and if all the
checks are passed, start the upgradation.

75
76
 The installer will ask for confirmation before executing root scripts as shown
above. Click Yes to start executing root scripts. This finishes the upgradation
of clusterware.

77
  Check the version of clusterware using this command to verify the successful
upgradation,

Section 2: Oracle RAC 12c database software installation

 The 12.1.0.2 database software should be installed before starting

upgradation. Navigate to database software stage location and start the
installer as below,

 Enter Oracle support credentials or uncheck the box and click next if you do
not want to receive any security updates.

78
 Select “Install database software Only” option from the available options and
click next,

 Perform a real application clusters database software installation by selecting

appropriate option as shown in the picture below

79
 Select all the nodes where you want to install Oracle RAC database software.
And also setup password less ssh connectivity by supplying the OS
credentials of the user performing the install. Then click next.

 Select desired language and the edition of the database to install and click
next,

80
 Specify the path to Oracle base and also give location for installing new
database software. Make sure the user performing the installation has
required permissions on the directory. Then click next,

81
 Assign appropriate OS user groups for operating system authentication.

 Oracle universal installer will now perform prerequisite checks for database
software installation. Below screen is displayed if all the checks are passed.
Click “Install” to start the installation. Ensure that all the prerequisites are met
before starting the installation.

82
 Run the root scripts in the specified order in all cluster nodes. Note that
database installer cannot run root scripts by itself as shown during grid
infrastructure upgrade in section 1. They have to be manually run as root
user.

 After the root scripts run successfully, click “Ok” to finish the database
software installation

83
Section 3: RAC database upgrade to 12c

 Start the database upgrade assistant (DBUA) from the newly installed oracle
home from section 2. The DBUA executable can be found under
“$ORACLE_HOME/bin”. On the initial screen, select the option “Upgrade
Oracle Database” and click next.

 This screen will list all the instances running in the server. Select the database
you want to upgrade and click “Next”.

Note: If the database you want to upgrade is not listed, check the contents of
“/etc/oratab” for an entry of the running instance.

84
 DBUA will run prerequisite checks display any errors or warnings if found,

 It can be seen from below screen that few components have given warnings
during prerequisite checks. As these components can be reconfigured or
upgraded after the database upgradation, click “Next” to ignore and proceed
to next step.

85
  You can specify the upgrade parallelism for speeding up the upgrade process.
The parallelism should be defined appropriately based on available resources
(No.of CPUs). Check the option to recompile invalid objects post upgradation.

If you want to run gather schema statistics or upgrade time zone, you can do it by
checking the respective boxes. Specify the location of diagnostic and audit file
destination or leave it to default location as shown. Custom sql scripts can be run by
giving their location under “Custom SQL Scripts” tab.

 If you wish to configure database express or register with enterprise manager

cloud control, check respective boxes otherwise click next to ignore and
proceed.

86
  Configure a backup and restore strategy for the database in case the
upgradation fails midway. If that happens this backup will be used to restore
the database to original state. DBUA can take a new rman backup of the
database before starting upgrade or if you have already taken the backup, you
can select “I have my own backup and restore strategy” and click next.

Note that if you select this option DBUA will not take any backup prior to starting
the upgrade.

87
 A summary screen will be displayed as below. In this screen we can see the
current database version and the version to which the database is going to be
upgraded, the status of the database instances, and any changes DBUA is
going to make to initialization parameters.

 DBUA will start the upgradation now and the progress can be seen on the
status bar. Depending on the amount of resources available and the size of
database, the upgradation can take very long time. Therefore patiently wait
for the upgrade to complete and keep checking the alert log for any errors.

88
  When the status bar reaches 100%, it means the upgrade has completed
successfully. Click on “Upgrade Results” to view the current version of the
database which is 12.1.0.2 and also the status of instances. If you wish to
revert back to 11g, click on “Restore Database” and proceed with the steps to
fully restore the database using the backup and restory strategy configured
earlier.

This completes the Upgradation of Oracle Grid Infrastructure and Oracle RAC
Database from 11.2.0.4 to 12.1.0.
89
Applying latest PSU to 12.1.0.2:
 Go to support.oracle.com and Search with “12.1.0.2 PSU”
 Check the latest version of the PSU and Download it for the respective OS
 Make sure the latest version of Opatch is downloaded with the help of the patch
6880880 and Unzip it in OH
 Shutdown all the services of the OH
 Unzip the Patch and run “opatch apply”

Applying Patches on RAC and Upgradations:

The methods used for Upgrades like Manual or DBUA are the same in RAC as well.
However the Process is a little different as we are dealing Nodes in the Cluster. Some key
points to take into consideration:

 The Services related to the Clusterware, Other Oracle Homes need to be properly
shutdown before Applying the patches
 Backup of Database (Using RMAN is Recommended), OCR, Voting Disk, Control File etc
is recommended
 Applying Patches or Upgrading should/will be done only from one of the Nodes and
the Updates to the other nodes are propagated automatically.
 10g to 11gR1 is almost the same however 10gR2 to 11gR2 is different because of Grid
Infrastructure

Applying 12.1.0.2 Patchset on 12.1.0.1:

The following are the steps that need to be performed for upgrading Oracle RAC from
12.1.0.1 to 12.1.0.2.

 Perform a Backup of the Database, OCR and Voting Disk

 Stop all the Services including the Database, Node Apps and the Clusterware
 First we need to Upgrade the Clusterware Software. Set the ORACLE_HOME to CRS
Home and Apply the 12.1.0.2 Patchset
 Bring up the Oracle Clusterware Services
 Before Upgrading the Database, Export ORACLE_HOME to RDBMS Oracle Home and
Apply the 12.1.0.2 Patchset.
 Now that the Software has been Upgraded to 12.1.0.2, Upgrade the Database to 12.1.0.2
with the help of Manual Method(catupgrd.sql) or DBUA(Database Upgrade Assistant)d

90
 Clusterware Administration:
CRSCTL:

CRSCTL is one of the key tools to manager Oracle Clusterware using which the Clusterware
processes and resources can be stopped and started. It is also useful in debugging the
Clusterware related issues and find the root cause quickly. The following are some of the
important CRSCTL syntaxes and the uses:

crsctl check crs - To Check the viability of the CRS stack:

crsctl check cssd - To check the viability of CSS

crsctl check crsd - To check the viability of CRS

crsctl check evmd - To check the viability of EVM

crsctl query css votedisk - To list the voting disks used by CSS

crsctl add css votedisk <path> - adds a new voting disk

crsctl delete css votedisk <path> - removes a voting disk

crsctl enable crs - enables startup for all CRS daemons

crsctl disable crs - disables startup for all CRS daemons

crsctl start crs - starts all CRS daemons

crsctl stop crs - stops all CRS daemons. Stops CRS resources in case of cluster.

crsctl start resources - starts CRS resources

crsctl stop resources - stops CRS resources

crsctl debug statedump evm - dumps state info for evm objects

crsctl debug statedump crs - dumps state info for crs objects

crsctl debug statedump css - dumps state info for css objects

crsctl debug trace css - dumps CSS in-memory tracing cache

crsctl debug trace crs - dumps CRS in-memory tracing cache

crsctl debug trace evm - dumps EVM in-memory tracing cache

crsctl query crs softwareversion [<nodename>] - lists the version of CRS software installed

crsctl query crs activeversion - lists the CRS software operating version

91
CRS_STAT:

It is used for monitoring the status of the Clusterware resources and the state of the
resources.

Find the command help : crs_stat -h

To find resource states : crs_stat -t

OCRCHECK:

It is used to find the status of the OCR file.

ocrcheck -help
Sample ocrcheck output.

OCRCONFIG:

This utility is used to manage the OCR File and can be used to perform the following:

 Restoring a corrupted OCR from a physical backup

 Add/Replace OCR copy with another OCR file
 To repair a damaged OCR with a new OCR version
 Export/Import OCR contents to a file

92
CLUVFY:Cluster Verification Utility (CLUVFY ) is the most important which is used during
Installation of Oracle RAC with the help of which, the DBAs can validate their installation
progress at different stages of Installation.

93
e.g.

The following syntax is used to validate the installation of Clusterware and the current state
to confirm whether we can proceed with the next step of Installation process

cluvfy stage -post crsinst -n raclinux1 -verbose

Workload Management
Load balancing is one of the significant features of Oracle RAC which improves the overall
performance. This is achieved by distributing the Workload efficiently between the available
Oracle RAC nodes.

Oracle performs workload distribution using connection balancing. In other words, all load
balancing activity is performed before a session establishes a connection. Once the
connection has been established to a specific instance, the session will communicate with the
same instance until either the session or the instance terminates.

Types of Connection Balancing:

 Client-side: Connection balancing involves the selection of a random instance by the

client process.
 Serverside: Connection balancing involves the selection of an instance by the
listener daemon on the server, based on metrics received by the listener from the
database.

Client-Side Connection Balancing:

RACDB =
(DESCRIPTION =
(LOAD_BALANCE = yes)
(ADDRESS = (PROTOCOL = TCP)(HOST = raclinux1-vip.localdomain)(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = raclinux2-vip.localdomain)(PORT = 1522))
(CONNECT_DATA =
(SERVICE_NAME = RACDB_MAIN)))

JDBC Example:
url="jdbc:oracle:thin:@(DESCRIPTION=(LOAD_BALANCE=on)
(ADDRESS=(PROTOCOL=TCP)(HOST=raclinux1-vip.localdomain)(PORT=1522))
(ADDRESS=(PROTOCOL=TCP)(HOST=raclinux2-vip.localdomain)(PORT=1522))
(CONNECT_DATA=(SERVICE_NAME=RACDB_MAIN)))"

When the LOAD_BALANCE parameter is set to ON, Oracle Net will select a node at
random from the list of addresses to establish a connection with the various listeners. No
load metrics are used for distributing work.

Client-side connection balancing works well if all instances are registered. Prior to Oracle
10.1, client-side connection balancing did not detect blocked, failed, or hung instances,
which could lead to skewed connection loads on each node. In Oracle 10.1, the combination
of Virtual IP (VIP) and Fast Application Notification (FAN) can be used to detect failed
instances.

94
One of the main disadvantages of this method is it is unaware of the existing load on the
nodes and therefore does not take account of how many other clients are currently
connected to the node or of the current workload on the node.

Server Side Listener Connection Load Balance:

The Server Side Listener Connection Load Balance feature allows the listener to route the
incoming user connections to the least-loaded node and then to the least loaded instance
using the information from the Load Balance Advisory. This feature balances the number of
active connections among multiple instances, which improves the connection performance.
To enable this feature, the remote_listener parameter must be configured in the RAC
instances.

In Oracle 10.2 and above, RAC includes a load balancing advisory which can supply
information to OCI, ODP.NET, and JDBC connection pools. Third-party applications can
subscribe to the load balancing advisory using the Oracle Notification Service (ONS). The
load balancing advisory monitors the current workload for each service in the instance and
creates FAN events.
LISTENER_RACDB = (DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = raclinux1-vip.localdomain)
(PORT = 1522))
(ADDRESS = (PROTOCOL = TCP)(HOST = raclinux2-vip.localdomain)
(PORT = 1522))
Transparent Application
(CONNECT_DATA = Failover:
(SERVICE_NAME = RACDB_MAIN)))
Transparent Application Failover (TAF) is an Oracle RAC feature that can completely mask
many failures from end-users, by preserving the state of their application and resuming any
work that had been in progress at the time of failure. TAF also provides developers with
tools to build applications that leverage TAF and make all failures, including those affecting
transactions, transparent to end-users.

95
Fast Connection Failover:
Starting with Oracle 10g, the TAF policy for JDBC thin and JDBC OCI drivers is supported
by setting up a new Fast Connection Failover (FCF) feature.

Services
Services are groups or classifications of applications that comprise business components that
extend and respond to application workloads. Examples of services are Accounts Payable
(AP), Customer Relationship Management (CRM), and so on. Services in RAC enable
continuous, uninterrupted database operations to support multiple services on more than
one instance.

Services can be created when you create a Real application Cluster (RAC) database. and
assign them to instances using the DBCA. After your RAC database is operational, you can
administer services and high availability using the DBCA, as well as Enterprise Manager
and SRVCTL.

SRVCTL:
You can create and administer services using SRVCTL utility, DBMS_SERVICE PL/SQL
package and Enterprise Manager.

Starting and Stopping Services with SRVCTL:

srvctl start service -d name [-s service_name_list] [-i inst_name] [-o start_options] [-c
connect_str | -q]

srvctl stop service -d name -s service_name_list [-i inst_name] [-o start_options] [-c
connect_str | -q]

96
Enabling and Disabling Services with SRVCTL:

By default, all services are enabled for automatic restart.

srvctl enable service -d db_name -s service_name_list [-i inst_name]

srvctl disable service -d db_name -s service_name_list [-i inst_name]

Relocating Services with SRVCTL:

You may have to relocate services sometimes from one instance to other

srvctl relocate service -d crm -s crm -i crm1 -t crm3

Statuses of Services with SRVCTL:

srvctl status service -d crm -s crm

SRVCTL can also be used for managing several resources like Node Apps, Instances,
Databases etc It is tightly integrated with Clusterware.

Start Services (Order: Nodeapps – ASM – Database)

 srvctl start nodeapps -n nodename

 srvctl start asm -n nodename
 srvctl start database -d dbname ( options are: srvctl start database -d dbname -o open
| -o mount | -o nomount)

Stop Services (Order: Database – ASM – Nodeapps)

 srvctl stop database -d dbname -o immediate

(options are: srvctl stop database -d dbname -o normal | -o transactional | -o
immediate | -o abort
 srvctl stop asm -n nodename
 srvctl stop nodeapps -n nodename
To check status and configurations Nodeapps:

srvctl status nodeapps -n nodename

srvctl config nodeapps -n nodename

ASM:
srvctl status asm -n nodename
srvctl config asm -n nodename

Database:

srvctl status database -d dbname

srvctl config database -d dbname (shows instances name, node and oracle home)
Instance:

97
srvctl status instance -d dbname -i instancename
srvctl start instance -d dbname -i instancename
srvctl stop instance -d dbname -i instancename

Administering OCR & Voting Disk

Though it is not mandatory it is obvious that every Oracle RAC customer implements
redundancy for OCR and Voting Disk. Oracle recommends that you select the option to
configure multiple voting disks during Oracle Clusterware installation to improve
availability. If necessary, you can dynamically add voting disks after you complete the
Oracle Clusterware installation process.

Administering Voting Disks

Oracle recommends that you back up your voting disk after the initial cluster creation and
after you complete any node addition or deletion procedures.

Determine the current voting disk by issuing the following command:

crsctl query css votedisk

Issue the dd command to back up a voting disk, as appropriate. In the following examples,
voting_disk_name is the name of the active voting disk and backup_file_name is the name
of the file to which you want to back up the voting disk contents:

dd if=voting_disk_name of=backup_file_name

Recovering Voting Disks

dd if=backup_file_name of=voting_disk_name

If you have multiple voting disks, then you can remove the voting disks and add them back
into your environment using the following commands, where path is the complete path of
the location where the voting disk resides

crsctl delete css votedisk path

crsctl add css votedisk path

Adding, Moving, or Deleting Voting Disks

You can add, move, and remove voting disks after Oracle Clusterware has been installed
using the following commands:

To retrieve the current voting disk used by CSS, issue the following command:

crsctl query css votedisk

To add a voting disk, issue the following command as the root user, replacing the path
variable with the fully qualified path name for the voting disk you want to add:

98
crsctl add css votedisk path -force

To move a voting disk, issue the following commands as the root user, replacing the path
variable with the fully qualified path name for the voting disk you want to move:

crsctl delete css votedisk path -force

crsctl add css votedisk path -force

Administering the Oracle Cluster Registry

 Automatic backups—Oracle Clusterware automatically creates OCR backups every
four hours. At any one time, Oracle Database always retains the last three backup
copies of the OCR. The CRSD process that creates the backups also creates and retains
an OCR backup for each full day and at the end of each week. You cannot customize the
backup frequencies or the number of files that Oracle Database retains.

 Manual backups—This feature has been introduced from 11g. Use the ocrconfig -
manualbackup command to force Oracle Clusterware to perform a backup of the OCR
at any time, rather than wait for the automatic backup that occurs at 4-hour intervals.
The -manualbackup option is especially useful when you to need to obtain a binary
backup on demand, such as before you make changes to the OCR.

Adding an Oracle Cluster Registry

You can add an OCR location after an upgrade or after completing the Oracle Clusterware
installation. If you already mirror the OCR, then you do not need to add an OCR location;
Oracle Database automatically manages two OCRs when it mirrors the OCR. Oracle
Clusterware environments do not support more than two OCRs: a primary OCR and a
secondary OCR.

Note:

If the OCR resides on a cluster file system file or if the OCR is on a network file system, then
create a dummy file before performing the procedures in this section.

As the root user, issue the following command to add an OCR location using either
destination_file or disk to designate the target location of the additional OCR:

ocrconfig -replace ocr destination_file or disk

Run the following command to add an OCR mirror location using either destination_file or
disk to designate the target location of the additional OCR:

ocrconfig -replace ocrmirror destination_file or disk

Run the following command on any node in the cluster to remove the OCR device:

ocrconfig -replace ocr

99
Run the following command on any node in the cluster to remove the mirrored OCR device:

ocrconfig -replace ocrmirror

Restoring the Oracle Cluster Registry on Linux or UNIX Systems

ocrconfig -restore file_name

OCR Import/Export
The OCR Import/Export commands can also be used as an alternative for backup and
recovery of OCR

ocrconfig -import file_name

ocrconfig -export file_name

ASM supports OCR and Voting Disk files from 11gR2. The OCR is striped and mirrored (if
we have a redundancy other than external), similar as ordinary Database Files are. So we can
now leverage the mirroring capabilities of ASM to mirror the OCR also, without having to
use multiple RAW devices for that purpose only. The Voting Disk (or Voting File, as it is
now also referred to) is not striped but put as a whole on ASM Disks – if we use a
redundancy of normal on the Diskgroup, 3 Voting Files are placed, each on one ASM Disk.
This is a concern, if our ASM Diskgroups consist of only 2 ASM Disks! Therefore, the new
quorum failgroup clause was introduced:

create diskgroup data normal redundancy

failgroup fg1 disk 'ORCL:ASMDISK1'
failgroup fg2 disk 'ORCL:ASMDISK2'
quorum failgroup fg3 disk 'ORCL:ASMDISK3'
attribute 'compatible.asm' = '11.2.0.0.0';

The failgroup fg3 above needs only one small Disk to keep one Mirror of the Voting File. fg1
and fg2 will contain each one Voting File and all the other stripes of the Database Area as
well, but fg3 will only get that one Voting File.

Voting File is no longer supported to be taken a manual backup of it with dd. Instead, the
Voting File gets backed up automatically into the OCR.

Configuring Initialization Parameters for an Oracle RAC Database

In Oracle RAC, each instance can have seperate parameter file or all the instances can have a
single parameter file. Oralce RAC parameters can be classified into 3 categories:

Parameters that Must Have Identical Settings on All Instances

 ACTIVE_INSTANCE_COUNT
 CLUSTER_DATABASE
 CLUSTER_DATABASE_INSTANCES

100
  COMPATIBLE
 CONTROL_FILES
 DB_BLOCK_SIZE
 DB_DOMAIN
 DB_FILES
 DB_NAME
 DB_RECOVERY_FILE_DEST
 DB_RECOVERY_FILE_DEST_SIZE
 DB_UNIQUE_NAME
 INSTANCE_TYPE (RDBMS or ASM)
 PARALLEL_MAX_SERVERS
 REMOTE_LOGIN_PASSWORDFILE
 RESULT_CACHE_MAX_SIZE
 UNDO_MANAGEMENT

Parameters that Must Have Unique Settings on All Instances

 INSTANCE_NUMBER
 THREAD
 ROLLBACK_SEGMENTS
 UNDO_TABLESPACE
 INSTANCE_NAME

Parameters that Should Have Identical Settings on All Instances

 ARCHIVE_LAG_TARGET
 CONTROL_MANAGEMENT_PACK_ACCESS
 LICENSE_MAX_USERS
 LOG_ARCHIVE_FORMAT
 SPFILE
 UNDO_RETENTION

Flash Recovery Area:

Oracle recommends that you enable a flash recovery area to simplify your backup
management and deally, the flash recovery area should be large enough to contain all the
following files:

 A copy of all datafiles

 Incremental backups
 Online redo logs
 Archived redo log files that have not yet been backed up
 Control files and control file copies
 Autobackups of the control file and database initialization parameter file

101
DB_RECOVERY_FILE_DEST and DB_RECOVERY_FILE_DEST_SIZE parameters to the
same values on all instances. Recommended Configuration is using ASM with different Disk
Groups other than once being used for Data Files.

Archive Log Mode in Oracle RAC:

In an Oracle RAC environment, each instance will have its own set of online redolog files
known as a thread. Each Oracle instance will use its set (group) of online redologs in a
circular manner. Once an online redolog fills, Oracle moves to the next one. If the database is
in "Archive Log Mode", Oracle will make a copy of the online redo log before it gets reused.
A thread must contain at least two online redologs (or online redolog groups).

An instance does not need access to the archived redo log files from a different instance
except when performing backup or recovery operations.

export ORACLE_SID=prod1
sqlplus '/as sysdba'
SQL> alter system set cluster_database=false scope=spfile sid='prod1';
System altered.
SQL> exit
srvctl stop database -d prod
sqlplus '/as sysdba'
SQL> startup mount
SQL> alter database archivelog;
Database altered.
SQL> alter system set cluster_database=true scope=spfile sid='prod1';
System altered.
SQL> shutdown immediate
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.
srvctl start database -d prod
sqlplus '/as sysdba'
SQL> archive log list;

102
 Troubleshooting Oracle RAC:
Before understanding different techniques of troubleshooting Oracle RAC Issues, it is
important to understand some basic commands to monitor services and locations of some
important files.

Find status of Clusterware Stack:

./crsctl check crs

Find OCR Locations:

ocrcheck

Find Voting Disk Locations:

./crsctl query css votedisk

or

check the output of "ocrdump"

Check status of all resources ( Nodeapps, ASM, Database, RAC Services):

crs_stat -t

A RAC DBA might possibly face several issues which might be related to Clusterware Stack,
Resources, OCR & Voting Disk etc.

Debugging Resources:
In the below example while trying to start the resources, if we get the issue like below:

CRS-0215: Could not start resource 'ora.prod2.vip'

We can debug any resources with the help of crsctl command as below:

./crsctl debug log res "ora.prod2.vip:2"

":2" denotes level of debugging and can be in the range of 1 to 5.

Checking the log files:

$CRS_HOME/log/<hostname>

Debugging Components:
We can also debug the Clusterware components i.e. CRS, EVM, OCSS etc

crsctl debug log crs "CRSD:1"

Diagnostics Collection Script

Every time an Oracle Clusterware error occurs, you should use run the diagcollection.pl
script to collect diagnostic information from Oracle Clusterware in trace files. The

103
diagnostics provide additional information so Oracle Support can resolve problems. Run
this script from the following location:

CRS_home/bin/diagcollection.pl

Oracle Clusterware Alerts

Oracle Clusterware posts alert messages when important events occur. The alerts contains
information about the entire Clusterware stack. For example events related to EVM, CRS or
OCSS etc

CRS_home/log/hostname/alerthostname.log

Important Log File Locations:

104
 Troubleshooting OCR & VOTING DISK:
Failed or corrupted Vote Disks:

 Implement multiple copies of the vote disk on different disk volumes to eliminate
Single Point of Failure.
 Make sure that you take regular backups for vote disks with the dd utility
(Unix/Linux) or copy utility (Windows)
 The Same dd command can be used in Linux/Unix to restore the backup Voting
Disk
 Without a backup of the vote disk, CRS should be reinstalled.

Failed or corrupted OCR:

Both OCR and Voting Disks are always recommended to be kept in different disks to avoid
single point of failure.

 Use the OCRCONFIG utility to perform recovery

 Export the OCR whenever changes are made to cluster resources

ocrconfig –showbackup

ocrconfig –restore backup10.ocr

Handling Node Eviction Issues:

The Oracle Clusterware is designed to perform a node eviction by removing one or more
nodes from the cluster if some critical problem is detected. A critical problem could be a
node not responding via a network heartbeat, a node not responding via a disk heartbeat, a
hung or severely degraded machine, or a hung ocssd.bin process.

Important Log Files:

 Clusterware alert log in $GRID_HOME>/log/nodename

 The cssdagent log(s) in
$GRID_HOME/log/nodename/agent/ohasd/oracssdagent_root
 The cssdmonitor log(s) in
$GRID_HOME/log/nodename/agent/ohasd/oracssdmonitor_root
 The ocssd log(s) in $GRID_HOME/log//cssd
 The lastgasp log(s) in /etc/oracle/lastgasp or /var/opt/oracle/lastgasp
 IPD/OS or OS Watcher data
 ‘opatch lsinventory -detail’ output for the GRID home

OS Messages files:

 Linux: /var/log/messages
 Sun: /var/adm/messages
 HP-UX: /var/adm/syslog/syslog.log
 IBM: /bin/errpt -a > messages.out

105
Common Causes:

 Network failure or latency between nodes. It would take 30 consecutive missed

checkins (by default – determined by the CSS misscount) to cause a node eviction.
 Problems writing to or reading from the CSS voting disk. If the node cannot perform
a disk heartbeat to the majority of its voting files, then the node will be evicted.
 A member kill escalation. For example, database LMON process may request CSS to
remove an instance from the cluster via the instance eviction mechanism. If this times
out it could escalate to a node kill.
 An unexpected failure of the OCSSD process, this can be caused by any of the above
issues or something else.
 An Oracle bug

OS Watcher:
OS Watcher (OSW) is a collection of UNIX shell scripts intended to collect and archive
operating system and network metrics to aid support in diagnosing performance issues.
OSW operates as a set of background processes on the server and gathers OS data on a
regular basis, invoking such Unix utilities as vmstat, netstat and iostat.

OS watcher can be downloaded from My Oracle Support " Doc ID: 301137.1"

Once the tar file downloaded is extracted, from the the extracted directory, run

e.g. Configure OS Watcher to take snapshots with the system utilities at every 5 minutes for
the last 24 hours.

nohup ./startOSWbb.sh 300 24 &

106
 12c RAC New Features:
Oracle Flex Cluster 12c

Oracle Clusterware, in the original design, is a tightly coupled clustering solution which
requires shared storage and shared network between all servers in the cluster. This is a
requirement when Oracle Clusterware is installed for Oracle databases - RAC and single
instance. This tightly coupled cluster is required to preserve data integrity of multi-instance,
shared cache clustered databases. Oracle 12c introduces an innovative clustering solution -
Oracle Flex Cluster which relaxes the tightly coupled nature of Oracle Clusterware and
permits the collocation of applications and the RDBMS in a common Oracle Flex Cluster. In
the Oracle Flex Cluster, the database tier, composed of 'HUB' servers, preserves the tightly
coupled configuration, while the application tier, composed of 'LEAF' servers, is loosely
coupled and does not require shared storage and peer-to-peer communication with other
servers in the cluster. This innovative tightly coupled/loosely coupled clustering solution
provides far greater scalability without compromising the availability and resource
management expected in high performance, and very large scale consolidations.

Business Continuity and High Availability:

Oracle Flex Clusters extends the availability and management features well known in Oracle
Real Application Clusters to the application tier. In the initial 12c release, Oracle Flex Cluster
is designed for collocation of applications and the RDBMS in a common cluster using the
HUB-LEAF deployment model described above.

The collocation of applications and the database in a common cluster offers far greater
availability and ease of management between application components and the database,
where start, stop ordered dependencies, and placement may be defined and automated.
Specifically, applications may depend on components outside of the respective application
management domain. Customers either execute the start up of these various components
manually, or, they maintain cumbersome scripts in an attempt to automate. For example, a
PeopleSoft application may have a WebLogic component and a Tuxedo component, each of
which are managed within their respective domains. Oracle Flex Cluster allows you to
manage these components collectively while maintaining secure, isolation management
between the components. And if the database is not up, the dependency model can be
extended between the application and database tiers allowing the application to pull up the
database or, if the application tier is down, the database startup would start the application
components on the correct servers in the correct order. The entire process, application tier
start up with correct startup order and placement and database/application tier pull-up - is
fully automated.

Oracle Flex Cluster is compatible with the Oracle Clusterware Agents. These Agents,
developed by Oracle Clusterware engineering and the respective application engineering
groups, are fully integrated to provide high availability (HA) and online relocation services
for applications running in the Oracle Flex Cluster. The dependencies described above are
implicitly configured by Oracle and can extend between the application and the database.

107
The Agents allow for applications to define start and stop dependencies on database services
or the databases directly

The above figure illustrates the Oracle Flex Cluster coupled with no shared storage or peer
cluster. A LEAF node has a loose communication association with a single HUB node in the
cluster.

Oracle Flex Cluster is the innovative clustering solution to preserve and protect business
critical functions and insures continuous, automated availability for application and
database services.

Scalability and Agility:

Oracle VM and Bare Metal

As noted earlier, Oracle Flex Cluster is designed for optimal scalability where shared storage
and peerto-peer communication is restricted to the HUB tier. This design supports very
large scale deployments of 100s of nodes. Another innovation introduced with Oracle Flex
Cluster 12c is the ability to deploy Oracle VM guests and Bare Metal servers in the same
cluster. Earlier releases of Oracle Clusterware support Oracle Clusterware deployments in
Oracle VM environments for Oracle RAC and applications, however, all servers in the
cluster are expected to be Oracle VM guest. Oracle Flex Cluster, however, allows the
application/LEAF servers to be deployed as Oracle VMs while the HUB servers are Bare
Metal. This deployment model allows for rapid application provisioning while preserving
HUB tier performance requirements. In addition, the Oracle Flex Cluster HA and policy
management capabilities are complimentary to Oracle VM. Oracle Flex Cluster Agents
rapidly detect VM hangs or infrastructure failures and provide hot standby failover of the
application to another VM in the cluster. The start/stop dependencies, ordering and
placement driven through the Oracle Flex Cluster policy engine guarantees ordered start

108
and stop of application components and a rich placement configuration for attraction,
dispersion and hard, weak, pull-up and exclusion ordering for Bare Metal and Oracle VM
servers in the Oracle Flex Cluster.

The online resource relocation service of Oracle Flex Cluster can online relocate applications
or application components running inside Oracle VM guests to a hot standby Oracle VM
guest in the cluster. This functionality offers fine-grain control of application and resource
management within a VM and compliments the course-grained control of Oracle VM Live
Migration.

The complimentary power of Oracle VM and Oracle Flex Cluster allow for very large scale
deployment and agile resource management of applications running in Oracle VM guests
and Bare Metal servers.

Cost-effective Workload Management:

Oracle Clusterware has long provided functionality to efficiently manage concurrent

workloads. Online workload management has become more important as customers deploy
dense, multitenant configurations competing for shared resources. Oracle Clusterware
introduced server pool partitioning in 11g release 2, where servers in the cluster may be
logically partitioned for resource placement and to manage online server allocation. Server
pools may be re-partitioned online to accommodate fluctuating demand for critical
workloads. Originally, server pools were restricted to a set of basic attributes characterizing
servers as belonging to a pool, with no way to distinguish between types of servers or
specific compute attributes of servers within a pool. All servers were considered to be equal
in relation to their processors, physical memory, and other characteristics.

Server Categorization

Oracle Clusterware 12c introduces Server Categories, where Oracle Clusterware will
dynamically discover the computing attributes of a server, such as processor types, memory
and other distinguishing system features and allow users to organize servers into server
pools based on these categories. When SERVER_CATEGORY is defined in place of
SERVER_NAMES, server membership in a pool is restricted to servers with attributes that
satisfy the category. Users can define their own characteristic of servers for categorization in
addition to the discoverable attributes. Oracle Flex Cluster 12c provides two default named
categories, HUB and LEAF for the specific server roles configured. Use of server categories
provides an efficient, automated mechanism to allocate compute resources to server pools
based on pre-defined server attributes.

Cluster Configuration Policy Sets

The use of server pools and server categorization requires the cluster to be configured as a
policymanaged cluster. Prior to Oracle Clusterware 12c, server pool changes were manually
executed by the end-user based on fluctuating demand or as a recommendation from the
Oracle Database Quality of Service Management framework (QoS Management). Oracle
Clusterware 12c allows users to define a cluster configuration policy for each server pool

109
defined in the system. These policies collectively represent the cluster configuration policy
set. An administrator can set different policies to be in effect at different dates or times of the
day in accordance with business needs and the system demands. The policy in effect drives
the cluster topology as a whole based on server pool requirements for the target workload.
Cluster configuration policy sets provide dynamic management of cluster polices across the
cluster.

Cluster configuration policy sets and policy based management:

 Enable online server reallocation to satisfy workload capacity requirements

 Guarantees the allocation of required resource for critical work as defined by the policy
 Ensures isolation where necessary, so that dedicated servers are provisioned in the
cluster for applications and databases
 Enables policies to be configured to change pools in accordance with changing business
needs or application demand

Oracle Database Quality of Service Management

Oracle Database QoS Management1 is a full Oracle stack development effort introduced in
11gR2 to provide effective runtime database management of datacenter SLAs by ensuring
when there are sufficient compute and memory resources to meet all objectives these
resources are properly allocated. Should demand or server availability change where
capacity is over-subscribed, Oracle Database QoS Management insure that the most business
critical SLAs are preserved at the cost of less critical work. Oracle Clusterware 12c cluster
configuration policy sets are Oracle Database QoS Management aware. If you create policies
with Oracle Database QoS Management the Oracle Clusterware 12c cluster configuration
policy sets are disabled in deference to the Oracle Database QoS Management server pool
directive overrides.

Predictability

An important element in consolidated environments, where resources and resource use

fluctuate on demand, is the ability to predict the placement of resources after online
relocations or resource faults. Understanding the impact of changes to the environment is
important. Oracle Clusterware 12c provides a set of evaluation commands to determine the
impact of certain operations before the respective operation is executed. Administrators can
use the Oracle Clusterware 12c What-If command to predict Oracle Clusterware's response
to a hypothetical planned or unplanned event. Oracle Clusterware 12c has the ability to react
to events in the system and produce a response action plan.

This action plan consists of a series of resource state transitions and/or server pool
reconfigurations. What-If response plans are available for the following event categories:

110
 • Resources: start, stop, relocate, add, and modify

• Server Pools: add, remove, and modify

• Servers: add, remove and relocate

• Policies: change active policy

• Server Category: modify

Oracle Clusterware 12c What-If action plans help administrators determine failover capacity
for planned or unplanned events, confirm correct dependency definitions for configured
resources and validate policy set deployments. It is an essential tool in automated workload
management.

Standardized Deployment and System Management:

The key management strength of Oracle Clusterware is the efficient management of

workload in the cluster. Oracle Clusterware supports two types of workload management
configurations:

1. Policy-managed which allows for flexible management by abstracting resource allocations

(e.g. dynamically assigning resources to run in pools of servers in the cluster)

2. Administrator-managed which is the traditional means of managing resources where

resources are statically assigned to servers in the cluster and which is an inflexible, manual
management configuration

Oracle Clusterware supports both deployment configurations simultaneously; however,

servers allocated to Policy Management are not available to be Administrator-managed.
Support for both deployment configurations facilitates the ease of migration from
Administrator-managed to Policymanaged providing optimal flexibility required for
consolidation.

Policy Management and server pools offer a standardized deployment model for database
and application clustering. As stated, Oracle Flex Cluster 12c provides HUB server pools for
the database tiers and LEAF server pools for applications. Oracle Clusterware has a common
command line interface to manage resource in both tiers either by a single GRID
Administrator or, by secure, roleseparated administrators for each tier. The interface is the
standard Oracle Clusterware interface for all cluster resources.

Oracle Clusterware is fully integrated with Oracle Cloud Control. Oracle Clusterware
configured resources - cluster, local, network, application, database, VM and storage are all
exposed as Enterprise Managed targets which provides a single management 'pane of glass'
for all tiers managed by Oracle Clusterware. All Oracle Clusterware management operations
are available through the Oracle Cloud Control GUI.

111
 11gR2 vs 12c Database differences
S.No 11GR2
1 Single tenant architecture
2 Standard Oracle clusters can
accommodate between 32 to 64 nodes
3 No Pluggable Database concept in 11gR2
4 Single database structure.
5 Table space should be kept in read only
mode and data file should be offline to
move the data file.
6 Heap Tables(Non-Partitioned ) migration
is achieved.
7 Invisible indexes and virtual columns
feature Introduced.
8 Will create single Index on one column
9 All Redo data would be stored in UNDO
Tbs.
10 In 11g R2, SYSASM privilege was
introduced to perform ASM specific
operations.
11 SQL statements Execution at RMAN
prompt.
Eg: RMAN> sql 'select * from v$version '
;
12 Restore a table or partition using physical
backups is not possible.
13 Restricting PGA size is not possible. It
changes depends on load dynamically
14 Pre Upgrade script utlu112i.sql for any
issue fixes before Upgradations.
15 Standard sql file "catupgrd.sql" is used
for upgradation
12C
Multitenant architecture
Flex cluster and Flex ASM is designed to scale
up to 2000 nodes
Oracle 12c pluggable database plugging and
unplugging
Consolidation multiple 11gR2 databases into
Oracle 12c container database as PDB's
Online rename and relocation of an active data
file and additionally data files can be migrated
between storages e.g. non-ASM to ASM and
vice versa.
Online migration of table partition or sub-
partition. a table partition or sub-partition can
be moved to a different table space online or
offline.
Along with Invisible indexes, Invisible column
concepts has been introduced in Oracle 12c R1.
Will create multiple indexes on the same
column or set columns with different Index
types.( only one index applicable at a time.)
temporary undo feature in 12c R1 to reduce the
load the actual UNDO table space and less redo
generation.
SYSBACKUP privilege has been introduced in
12c to execute backup and recovery commands
in Recovery Manager (RMAN).
SQL statements Execution at RMAN prompt.
Eg: RMAN> select * from v$version ;
you can recover a particular table or partition to
a point-in-time or SCN from RMAN backups
Restricting PGA size is possible.(
PGA_AGGREGATE_LIMIT)
Preupgrd.sql replaces legacy utlu121i.sql script
and It provides fix up scripts to easy the process
of fixing issues in Upgradations.
catctl.pl (parallel-upgrade utility) introduced in
12C to improve the upgradation time by
parallel mode of execution.
112
16 Restore/Recover data files over the
network is not possible.
17 There is no feature available in
Datapump to turn off redo generation.
18 AWR,ASH,ADDM will not work
properly when DB was hung completely
and un responsive.
19 Gathering Database statistics on single
table at a time.
Restore/Recover data files over the network
using SERVICE NAME.
Turn off redo log generation while import
operation in
Datapump(TRANSFORM=DISABLE_ARCHI
VE_LOGGING:Y)
Real-time ADDM analysis(Emergency
Monitoring Option in 12c Cloud Control)
Gathering Database statistics concurrently on
multiple tables

11gR2 vs 12c RAC differences

S.No 11GR2 12C

1 Each Oracle RAC Instance having ASM
instance on same server for storage
management.
2 Logical Corruptions can be discovered while
RMAN Backup jobs are ran.
3 Individual Password file for each Instance.
4 IPv6 was supported in Oracle database 11gr2
but was only available for a standalone
database
5 No Multiple SCAN’s Per Subnet mask, per
cluster.
6 Standard Patching process
7 Manually need to run root.sh script on all
nodes one by one.
In 12c,with Oracle Flex ASM and cluster, reduce
overhead on Database Server with ASM Instance
running remotely and avoids single point of
failure .
ASM Disk Scrubbing feature to identify the
logical corruptions easily.
Shared password file for all RAC instances in
cluster .
In 12c Database, clients can also now connect to
database in RAC environments, using the IPv6
protocol.
Multiple SCAN’s Per Subnet mask, per cluster for
redundancy.
GHCTL utility for better patching.
UI Auto runs the root.sh script on all nodes

Flex Cluster & Flex ASM in 12c

Flex Cluster and Flex ASM:

From 12c Release onwards Oracle introduces two features called "Flex ASM" and "Flex
Cluster" that gives support to the demand requirements on Cloud Computing oriented
environments.
In previous releases, Oracle clusters can accommodate between 32 to 64 nodes. But in this
new release Oracle 12c, new feature Flex cluster is designed to scale up to 2000 nodes.
Mainly, multitenant architecture of flex cluster can be used to efficiently extend the high

113
availability frame work of oracle clusterware into the application tier. With flex cluster, we
can manage multiple databases and applications running in a single cluster.

Flex cluster:

Oracle 12c offers two types of Clusterware configuration setups: Standard Cluster and Flex
Cluster. In a standard Oracle Clusterware configuration, all nodes in the cluster tightly
coupled through private network and have direct read/write access to shared storage. A
Flex Cluster, on the other hand, designed over HUB-LEAF topology to scale the Cluster to
large number of nodes, up to 2000 nodes. Hub nodes are very much similar to the standard
cluster architecture, they are tightly coupled through a private interconnect and have direct
access to shared storage. On the contrary, Leaf nodes are very light weight servers, not
connected to other leaf nodes, and have no direct access to the storage. A Hub node can have
multiple leaf nodes, but, a node leaf can’t be part of multiple of Hub nodes. In a nutshell, the
prime purpose of an Oracle Flex Cluster is to provide the combined database and
application tiers together in a single clustering environment.

Hub Node:

Hub Nodes are similar to Oracle Grid Infrastructure nodes in an Oracle Clusterware
standard Cluster configuration. They are tightly connected and have direct accesses to
shared storage.

Leaf Node:

Leaf Nodes are different from standard Oracle Grid Infrastructure nodes, and they do not
require direct access to shared storage, but instead request data through Hub Nodes. Leaf
Nodes must be members of a cluster that includes at least one Hub Node.

Figure 1: Flex Cluster Architecture

Eg: Get the cluster mode status

114
$ crsctl get cluster mode status

Cluster is running in “flex” mode

If the cluster is running in standard mode, we can easily change from standard to flex mode
using CRSCTL utility.

Please note before converting standard cluster to Flex Cluster:

1. Involves cluster downtime

2. Movement of standard cluster to flex cluster is supported, but not vice verse.
3. Involves configuration of GNS with fixed IP, because flex cluster
mode doesn't support static DNS IPs.

Standard Cluster to Flex Cluster change:

Step 1: To check the current mode of the cluster:

# crsctl get cluster mode status

cluster is running in “standard” mode

Step 2: Run the following command to ensure that the Grid Naming Service (GNS) is
configured with a fixed VIP:

$ srvctl config gns

This procedure cannot succeed unless GNS is configured with a fixed VIP. If there is no
GNS, then as root create one as follows:

NOTE: before executing the following step, please ensure to make the GNS VIP IP entry in
the DNS server.

$ srvctl add gns -vip vip_name | ip_address

Run the following command as root to start GNS:

$ srvctl start gns

Step 3: Before changing the cluster mode, oracle flex ASM has to be enabled in cluster.
Enable the flex ASM using ASMCA.

$ asmcmd showclustermode

ASM cluster: Flex mode disabled

Step 4: Now, using ASMCA you can convert oracle ASM configuration to oracle FLEX ASM.
Before that we need to create password file in shared disk group using ORAPWD utility.

$ ORAPWD file=+OCR/asm.pwd password=***** entries=10

115
$ asmca

In ASM instances page you can select the “convert to oracle flex asm” dialogue box.

In this tab, select the instances to perform the operations and then click the “convert to
Oracle Flex ASM” dialogue box which is highlighted in RED. It will automatically open one
more tab, which specifies the listener port and network interface for the conversion to flex
ASM.

Select the network to be used for ASM network and then click OK button.

116
After that, run the above script as privileged user in local node for complete ASM
conversion. This script should take some time for converting ASM configuration to flex
ASM.

$ su – grid

Run the script “converttoFlexASM.sh” as grid user.

Step 4: Now we can check the flex ASM mode

$ asmcmd showclustermode

ASM Cluster: Flex mode enabled

Now, we can change the mode of the cluster to FLEX cluster as ‘ROOT’ user.

# crsctl set cluster mode flex

Cluster mode set to “flex”

Changes to cluster mode will be effective after cluster service restart.

# crsctl stop crs

And then start the cluster in each node

# crsctl start crs

Status of cluster mode is “FLEX”

Flex ASM:

The Next generation ASM(Automatic Storage Management ) has new name i.e Flex ASM.

As normal ASM instance will run on each node with Flex configuration to support pre 12c
database. The compatibility parameter of ASM disk group is used to manage the
compatibility of between and among database instances. Advantage of this approach is that
if an Oracle 12c database instance loses connectivity with an ASM instance, then the

117
database connections will failover to another ASM instance on a different server. This
failover is achieved by setting the cardinality to all.

Why use Oracle Flex ASM

 Oracle Flex ASM supports larger LUN sizes for Oracle Database 12c clients.
 Maximum number of Disk Groups supported is 511.
 Flexibility to rename an ASM Disk in a Disk Group.
 ASM instance Patch-level verification
 Patch level verification is disabled during rolling patches
 Replicated Physical Metadata

Network enhancement in Oracle Flex ASM

In previous versions the cluster requires:

 A public network for client application access
 One or more private networks for inter-node communication within the cluster
including ASM traffic

Flex ASM adds the ASM network, which can be used for communication between ASM
and its clients to isolate and offload ASM trafficated Physical Metadata

118
Deploying Flex ASM

Below is screen shot from the Flex ASM Installer.

Choose the option "Use Oracle Flex ASM Staorage"

You can use the ASMCMD command to determine whether Oracle Flex ASM is
enabled.

$ asmcmd showclustermode

ASM cluster: Flex mode enabled

Use SRVCTL to determine the status of the instances in an Oracle Flex ASM
configuration.

$ srvctl status asm -detail

ASM is running on db12c1, db12c3

ASM is enabled

119
 Oracle 12c RAC Commands Changes

To check current ASM Mode:-

Command: asmcmd showclustermode
$ ./srvctl config asm
ASM cluster: Flex mode enabled

To check current status of the cluster :-

Command: crsctl get cluster mode status
Cluster is running in “flex” mode

OCR backup in ASM disk group:-

Command: ./ocrconfig -backuploc +DG_OCR

SRVCTL Commands:

SRVCTL with -eval option:-

Use: simulate running a command without making any changes to the system

Command: ./srvctl stop database –d TEST –eval

SRVCTL with -predict option:-

Use: The SRVCTL predict command evaluates the consequences of resource failure.
$ srvctl predict diskgroup -diskgroup DATA -verbose

Resource ora.DATA.dg will be stopped

Resource ora.DATA.ORAHOMEVOL.advm will be stopped

SRVCTL Start/Stop Options:

Command:

srvctl start database|instance –startoption NOMOUNT|MOUNT|OPEN

srvctl stop database|instance –stopoption NOMOUNT|MOUNT|OPEN

120