Oracle RAC & ASM

RAC Concept

What is Real Application Cluster?

Cluster Consists of multiple interconnected servers that appears to user as one single database,
RAC allow instance to reside from one server to another in case of Failure. The below Figure
explain the basic setup for Real Application Cluster. Another definition you should be aware of
Node, which is unit contain CPU and Memory called Server.

Before Start talking about Oracle RAC, what is the advantage of using Oracle RAC?

Scalability: multiple nodes allow cluster database to scale by single node database.
Availability: if any nodes failure other nodes in cluster, clients can continue working
without any effects.
Manageability: more than one database can be handled by oracle Cluster ware.

Osama Mustafa Page 1

 Oracle RAC & ASM

There are two kinds of cluster active/active and active/passive

Acitve/passive
In this setup usually we have two nodes, one of the nodes are available (active) and the other one
is not (passive), Oracle software should be on Shared storage , and only run on one node ( active
) in case of failure the cluster convert the shared storage to passive node in that case active node
now is passive and the passive node is now active.

There is number of Third Party software Corporation support this kind of cluster such as
Microsoft, Linux. Usually its called OS cluster.

Active/Active
In this kind of setup Oracle instance run concurrently on both server and client access to both
server at the same time, the instance should communicate with other node to ensure (heartbeat)
both server are available but in case any of server goes down the other server can handle the
workload, the benefits of active/active workload can be shared between servers.

Some other concept I would like to mention here, Shared Disk and Non Shared disk in oracle
Real Application Cluster Installation, theres two option when you install Oracle RAC Nothing
to Share means database is configured as number of nodes but each node has it own private disk
only thing shared will be is Database, option number two is shared everything all nodes access to
same data in this case if any of node failure no missing or loss of data.

When I am talking about Real Application Cluster I already mean High Availability (HA) which
is the same purpose of Oracle RAC ensure system or components is continuously operational for
long time. RAC can be complex and not easy to manage, design, install, configure and tune and
finally not to forget the license which is cost lot. The reason behind deploy RAC is different
from company to another some users implement RAC to ensure HA, Other user implement RAC
to manage workload. You can use alternative solution of RAC if you dont have that much of
money such Data Guard or third party Software which I already mention it.

Difference RAC & NON-RAC

I would like to mention that share the same concept, background Process in RAC found in Single
Node (PMON,SMON,DBWR,ARCn,CKPT) , in RAC system Each Instance has its own System
Global Area and Redo log , This is if we are talking about Structure but Oracle Cluster has some
Daemons Event Manager Daemon ( emvd ) ,Oracle Cluster synchronization Service Daemon
(cssd ) and finally cluster ready services daemon ( crsd ).notice both single instance and rac
instance can be managed by Enterprise Manager.

Osama Mustafa Page 2

 Oracle RAC & ASM

Another Component should be mention when talking about RAC Oracle Cluster Registry (OCR)
shared storage should be available for OCR the benefits of using OCR is Store cluster
configuration such as names, services and instance, Voting Disk also should be on shared storage
is required to determine the nodes that are currently available you can search for process
(CSSD).

Check OCR using crscheck or using crsctl query css votedisk

Node Application In Real Application Cluster

Is services automatically configured when On installing Phase, and enable by default when node
is booted.

Listener

Listener is process runs on each node and the purpose for this process is listening to incoming
connection from clients and redirected to RAC instance, since theres two type of connection
dedicated and shared listener is responsible for managing workload.

Oracle Notification Services ( ONS )

Started by the CRS cluster ware as part of the nodeapps. There is one ons daemon started per
clustered node. Receive a subset of published cluster ware events via the local evmd and
racgimon cluster ware daemons and forward those events to application subscribers and to the
local listeners.

Theres Two Type Of this kind of node app process Fast Application Notification ( FAN ) Which
is Used to notify application about any changes in configuration for example start/stop Services.
On other hand we Fast Connection Failover (FCF) The main purpose for this process is to
prevent any new connection for Failed nodes or instance.

Virtual IP ( VIP )

In Oracle 10gR1 Clients Can access to database Rac using VIP no More Public, VIP is Ip
address assign for each node belong to same Subnet mask For public, In case any Faiover VIP
Connect to Another Node.

Global Services Daemon ( GSD )

This process allows clients including SRVCTL and DBCA to execute administrative Command.

Osama Mustafa Page 3

 Oracle RAC & ASM

How to Check Oracle RAC Process is UP :

# ps ef | grep evmd

#ps ef | grep cssd

#ps ef | grep crsd

I can check ASM ( I will talk later about it )

#ps ef | grep asm

Also you Can Check http://support.oracle.com notes :

Script to Collect RAC Diagnostic Information (racdiag.sql) [ID 135714.1]

Which give you script to collect health Status for RAC

Or Another way

#Crsctl Check crs

Or using SRVCTL Command

Check Database Status

Srvctl status database d <Database-name>

Check status for Instance

Srvctl status instance d <Database-name> -I <Instance-1> ,<instance-2>

you can check nodeapps

Srvctl status nodeapps n <node-name>

Check All Resource

Crs_stat t

Osama Mustafa Page 4

 Oracle RAC & ASM

Trouble Shoot Oracle RAC:

OCR Failure
Vote Failure
Reboot Issue
Network Issue
Rac Resource Offline

To solve any Issue in Oracle RAC you need to Look at the logs first which is Located in

$CRS_HOME/log/nodename/

Also Some Useful http://support.oracle.com notes for troubleshoot:

Diagnosing CRS-215 Error When Starting Clusterware Resources [ID 781632.1]

Troubleshooting when srvctl can't start RAC instance, but sqlplus can start it [ID 844272.1]

Another issue you can face with Oracle RAC which is Resource offline, as mentioned before you
can check it by crs_stat t and usually the error CRS-0215 could not start resource %

You can try start this resourcr using srvctl command

Srvctl start nodeapps n <resourcre-name>

Is this not working you should go further more than this logs will tell what exactly you have.

Failed/Corrupted OCR and Voting Disk

RAC: Frequently Asked Questions [ID 220970.1]

How to Replace a Corrupt OCR Mirror File [ID 317628.1]

OCR usually backup automatically in oracle rac and can restore so the issue would be in voting
disk in Linux you can use dd command to backup voting disk, and for windows use copy
command.

And notice that if you dont have backup you need to reinstall CRS and this will cost you time
and Loss of money in same cases.

To restore OCR on RAC we use ocrconfig utility

Osama Mustafa Page 5

 Oracle RAC & ASM

OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) [ID 428681.1]

The backup path for OCR

$CRS_HOME/cdata/crs

#ocrconfig showbackup

#ocrconfig restore <backup-file-name>

Reboot issue in rac happened for different Cause maybe network,hardware,storage or even rac
process http://support.oracle.com note will help you to understand reboot

Troubleshooting 10g and 11.1 Clusterware Reboots [ID 265769.1]

Because of this you usually need to check matrix certification tab on support.oracle.com check
your network before starting setup process and health check for storage .

Monitor Oracle Real Application Cluster

Automatic Workload Repository (AWR)

o Collect Database statistics using Snapshot
o Check Database Performance And SQL time
o You can generate AWR using Command Line
@$ORACLE_HOME/rdbms/admin/awrrpt.sql
@$ORACLE_HOME/rdbms/admin/awrrpti.sql
Automatic Database Diagnostic Monitor (ADDM)
o Analyses data that captured by AWR
o Generate ADDM using Command Line

$ORACLE_HOME/rdbms/admin/addmrpt.sql
Enterprise manager
o Manage RAC database
o Display Performance as Graphical Interface
o Display Result for AWR,ADDM and ASH
Third Party Tools
o OS Command Such as top
o Spot Software
Active Session history (ASH)
o Help to solve simple performance issue.
o All active session will be generate every Second.

Osama Mustafa Page 6

 Oracle RAC & ASM

As conclusion you need to use EM to monitor oracle rac database and diagnose performance
Issue this will lead to identify the root of cause issue and try to solve it using one of the above
tools provided by oracle.

Automatic Storage management

First time ASM technology introduced in Oracle 9.0.1 By Oracle OMF (Oracle Managed Files)
Features which automated the creation for database files such as data files, Redo log, and control
file, you can manage table space without reference to filename, ASM Supported Single and RAC
databases.

ASM Cam manage many type of database files for example : data files , temp files , control files,
online redo logs, archive logs and RMAN backup sets, later in newer version it can supported
parameter files , flashback logs , and change tracking files.

Its designed to allow DBA (Database Administrator) to perform most disk management tasks; in
this case ASM has more advantage than LVM and RAID, Another Benefits for ASM For More
complexity Site which is managing large number of disk and you can do that by assign disk to
specific group, including to stripping and mirroring ASM Can Perform some other task such as
increasing reliability and performance By Default ASM Implements two-way mirroring to
ensure redundant copy of data therefore for critical data you can implement three way-mirroring,
another benefits for ASM which is provide mirroring at file level which give DBA more control.

Increasing Read and Write performance in ASM by strip mechanism by data across multiple
disk, and the stripes size control by database files, and as I mentioned earlier ASM Support
Single Instance and RAC database.

In Oracle Database 10gR1 ASM supported only Database 10gR1 check this example, if we have
database version 10.1.0.1 and ASM version 10.1.0.3 then ASM instance will revert to earlier
functionality , on the other hand if ASM version 10.1.0.2 and Database 10.1.0.3 then ASM will
continue using 10.1.0.2 functionality.

In Oracle 10gR2 this is totally different, ASM support different Database version, Oracle 10.2
ASM will support Oracle database 10.1 without losing any Functionality.

Osama Mustafa Page 7

 Oracle RAC & ASM

How to Install ASM

Before you start install any Oracle Products you have to look for Recommendation, In ASM
Oracle Corporation recommended to install ASM binary in separate Oracle Home Why? If we
are not going to use this approach patch & upgrade will be hard. Otherwise Separate Home will
be more flexibility to perform patching and Upgrading.

Another recommendation for Installing ASM Binary from Oracle which is when you Install
ASM for production environment use shared storage for the following reasons:

Upgrade and Patching Become Easier Only Once.

One Copy of binary Needs to be copy
Disk space for binary reduced.

Check https://oss.oracle.com to install the required package for ASM depend on your Platforms :

Oracleasm
Oracleasmlib
Oracleasm-support

To install them use the below command :

Rpm ivh <Package-Name>

When you finished installing the above package new device created for ASM called
/dev/oracleasm

And you can manage ASM using the below Command

[root@test root] # /etc/init.d/oracleasm <option>

Or

[root@test root] # service oracleasm <option>

The Options that can be used with the above command:

Start : Start ASM services

Stop : Stop ASM services
Status : Print the ASM Services Status
Configure : configure ASM
Createdisk: Create ASM disk
Osama Mustafa Page 8
 Oracle RAC & ASM

Scandisks : search for ASM disks

Querydisk: Query the current status of ASM disk
Deletedisk : delete an ASM disk
Restart : Restart ASM Services
Enable : Enable ASM services
Disable : Disable ASM services

All the above option will be described:

Start

[root@test root] # /etc/init.d/oracleasm start

stop

[root@test root] # /etc/init.d/oracleasm stop

Status

[root@test root] # /etc/init.d/oracleasm status

Checking if ASM is loaded: [OK]

Checking if /dev/oracleasm is mounted: [OK]

Configure

[root@test root] # /etc/init.d/oracleasm configure

# /etc/init.d/oracleasm configure

Configuring the Oracle ASM library driver.

This will configure the on-boot properties of the Oracle ASM library

driver. The following questions will determine whether the driver is

loaded on boot and what permissions it will have. The current values

will be shown in brackets ('[]'). Hitting without typing an

answer will keep that current value. Ctrl-C will abort.

Default user to own the driver interface []: oracle

Default group to own the driver interface []: oinstall

Start Oracle ASM library driver on boot (y/n) [n]: y

Osama Mustafa Page 9

 Oracle RAC & ASM

Scan for Oracle ASM disks on boot (y/n) [y]:

Writing Oracle ASM library driver configuration: done

Initializing the Oracle ASMLib driver: [ OK ]

Scanning the system for Oracle ASMLib disks: [ OK ]

Createdisk

Note On RAC (Real Application Cluster) these disk should be on Shared Storage using raw
device or logical partition, when you are using raw device make sure its identical on each node,
and regarding to htts://support.oracle.com note number 578455.1 raw device is un-supported in
11gR2, so another option are available now such as ASM,OCFS and OCFS2, also notice that
you need to create ASM disks only on one node, on the second one you should only able to run
scandisks attribute.

# /etc/init.d/oracleasm createdisk disk1 /dev/sdb1

Marking disk "disk1" as an ASM disk: [ OK ]

# /etc/init.d/oracleasm createdisk disk2 /dev/sdc1

Marking disk "disk2" as an ASM disk: [ OK ]

# /etc/init.d/oracleasm createdisk disk3 /dev/sdd1

Marking disk "disk3" as an ASM disk: [ OK ]

# /etc/init.d/oracleasm createdisk disk4 /dev/sde1

Marking disk "disk4" as an ASM disk: [ OK ]

Scandisks

After creating Disk you should run this command on the second node/nodes

# /etc/init.d/oracleasm scandisks

Scanning the system for Oracle ASMLib disks: [ OK ]

Osama Mustafa Page 10

 Oracle RAC & ASM

Listdisks

You need to verify the disk, this command show you the list of disks you have on ASM

# /etc/init.d/oracleasm listdisks

Disk1

Disk2

Disk3

Disk4

Querydisk

You can query the status disks using querydisk as the example :

# /etc/init.d/oracleasm querydisk disk1

Disk disk1 is a valid ASM disk

Deletedisk

You can delete existing disk using

# /etc/init.d/oracleasm deletedisk disk1

Removing ASM disk Disk1

ASM Administration Using ASMCMD

Asmcmd is new features in Oracle 10gr2 and above, ASMCMD utility allows you to inspect the
files within file system with very limited administration and modifications, and to use you should
export/set ORACLE_SID first to ASM instance as the below :

Export ORACLE_SID=+ASM1

By doing that new connection established, and the user should be member of SYSDBA group,
the below command supporting for ASMCMD

Cd
Du
Find

Osama Mustafa Page 11

 Oracle RAC & ASM

Help
Ls
Lsct
Lsdg
Mkalias
Mkdir
Pwd
Rm
Rmalias

The below describe some of the above command:

Cd

This commands same as Linux, allow you change current directory but this time within
ASMCMD

Cd <dir>

For example

Cd +DISKGROUP1

Du

Display the total amount of space

Du [dir]

For example

Du +DISKGROUP1

Used_MB Mirror_used_MB

500MB 700MB

Find

Allow you search inside file system,you can start with directoy which may include wildcards and
file name pattern

Find [-t type] <dir> <pattern>

For example :

Find +DISKGROUP1/orcl/redo*
Osama Mustafa Page 12
 Oracle RAC & ASM

+DISKGROUP1/orcl/redolog01.log

+DISKGROUP1/orcl/redolog02.log

+DISKGROUP1/orcl/redolog03.log

Help

Display summary of the commands available only in ASMCMD

Ls

List come or all files and their attributes, if you change the attribute output will be different each
time

Lsct

Lists information about current Oracle ASM clients.

lsct data

DB_Name Status Software_Version Compatible_version Instance_Name Disk_Group

+ASM CONNECTED 11.2.0.2.0 11.2.0.0.0 +ASM DATA

ASMVOL CONNECTED 11.2.0.2.0 11.2.0.0.0 +ASM DATA

Monitoring ASM
Theres two type of views to monitor ASM dynamic Performance view ( v$) and fixed view
(X$), notice both of these view are available for ASM and database instance.

Dynamic Performance Views

V$ASM_File

All ASM files belong to disk group mounted by ASM listed in this view.

V$ASM_DISK

List all disks discovered by ASM instance.

V$ASM_DISKGROUP

View lists all disk group discovered by ASM.

Osama Mustafa Page 13

 Oracle RAC & ASM

V$ASM_CLIENT

List all client database access to disk group.

V$ASM_ALIAS

List all aliases in all currently mounted disk group.

V$ASM_DISK_STAT

Same as V$ASM_DISK but if new disk added it will not discover it.

Fixed Views

View Name Usage

X$KFMTA ASM template
X$KFKID ASM Disk Performance
X$KFGRP ASM disk Group
X$KDSK ASM Disks
X$KGMG ASM Operations

Check ASM

#ps ef | grep asm

Using SQL PLUS

Sqlplus / as sysdba

Select name,path,state from v$ASM_DISK;

Osama Mustafa Page 14