<Insert Picture Here>

Oracle RAC OVM Templates OneCommand & the New DeployCluster Tool Saar Maoz RAC Development, Oracle
Updated: 18-JUL-2012

Agenda Oracle RAC & Oracle VM Overview Oracle RAC Oracle VM Templates NEW: DeployCluster tool Demo / Discussion
Standard 2 node Cluster Build OVM3: N-node Cluster Build using DeployCluster tool OVM2: Dom0 N-node Cluster Build Add / Remove Nodes / Instances Live Migration

What is Oracle Real Application Clusters ?

Allows a Shared Database to run on 2 or more nodes Requires

Shared Disk for Database Files Public and Private Networks Common OS version across nodes

Oracle RAC Architecture

//
public network

Node1

VIP1 Service Listener instance 1 ASM

Oracle Clusterware

VIP2 Service Listener instance 2 ASM

Oracle Clusterware

Node 2

VIPn Service Listener instance n ASM

Oracle Clusterware

Node n

Operating System

Operating System shared storage

Operating System

Managed by ASM RAW / Block Devices

Redo / Archive logs all instances Database / Control files OCR and Voting Disks

Oracle VM Templates
Rapid Application Deployment
E-Delivery

Download from Oracle

Pre-built, pre-configured VM Complete Clusterware, ASM, RAC installation Database 11g, Enterprise Manager dbControl
RAC
File

Save days or weeks in installation and configuration time

RAC
VM 1

RAC
VM 2 VM VM VM

Customize & Save as Golden Images

Oracle VM Server Pool

Oracle VM Servers Start-Up in Oracle VM Pool Import to Oracle VM Manager

NAS, SAN, iSCSI

VM

RAC OVM Templates - Availability Currently available for 11.2.0.3.2, 11.2.0.2.6, 11.2.0.3.0, 11.2.0.2.2, 11.2.0.2.0, 11.2.0.1.4, 11.2.0.1.2 ,11.1.0.7.6, 11.1.0.7.2 on Oracle Linux 32 and 64 bits
Download from e-delivery or My Oracle Support Note:1185244.1:
https://support.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id= 1185244.1

Templates are distributed as archive files containing two disk images

Oracle Linux 5 U4/U5/U6/U7/U8 & OL6U2 system disk OL5U8 and OL6U2 are OVMAPI enabled (Deploycluster) Oracle RAC install disk image (Clusterware, Database, ASM)
All homes updated to latest Bundle / CPU Patch level

Entire install is automated, with the user only providing minimal input parameters.

RAC OVM Templates - Delivery Mechanism 32 bit and 64bit versions Image files are built with sparse file support Inside each zip are tgz archives with the following files:
VM Config file (text file) Opatch lsinventory sample output README / PDFs for installing Disk 1 image file: Operating System Disk 2 image file: Oracle Software Includes Clusterware, ASM and RAC

Follow the instructions in provided PDF files or document for the new DeployCluster tool

RAC OVM Templates - Features

Builds production ready Oracle RAC (Cluster) in about 30 minutes 11.1: Simple 2-node interview, then manual add node 11.2 Templates further allow: OVM3: Network & N-node build using new DeployCluster tool OVM2: Automatic network & N-node build directly from Dom0 Mixing & matching any OS and Oracle disks Automatic add/remove any number of nodes or instances Run as root, oracle/grid users (w/sudo configured on build node only) Support for role separation or non-role separation (default) Options to modify environment before/after a build (Create DB, DB Console, ACFS filesystem, etc) Supports RAC One Node

Using The RAC OVM Templates Step 1 Download Template 32/64 11.1/11.2.0.1/11.2.0.2/11.2.0.3 Step 2 Register Template with Oracle VM Manager Step 3 Create 2+ VMs Step 4 Shared Storage
For Non Production: Create Shared Disk using OVM Manager Assigned Shared disk to all Guest VMs using OVM Manager For Production Deployment: Identify Physical Shared Disk OVM2: Update all Guest VMs vm.cfg files with location of shared disk OVM3: Attach physical disks using Oracle VM Manager

Step 5 Boot all VMs or use Deploycluster tool and skip Step 6 Step 6 Run OneCommand to Configure and Build RAC database

RAC on OVM Deployment Configuration Two types of deployments

Production Oracle RAC OVM Configurations [Supported]
Each RAC node must be deployed on a separate physical server for production environments Shared database disks must be on physical disks

Test Oracle RAC OVM Configurations

Above requirements are relaxed

Templates support both configurations, each has its own PDF to help with steps

10

Oracle RAC on Oracle VM

Production Deployment

Service Listener instance 1 Oracle ASM

Oracle Clusterware

Service Listener instance 2 Oracle ASM

Oracle Clusterware

Dom0 OS DOM 0

GuestSystem OS Operating Guest VM

GuestSystem OS Operating Guest VM

Dom0 OS DOM 0

DOM-0

Hypervisor
X86/64 Bare-Metal Server

Hypervisor

DOM-0

X86/64 Bare-Metal Server

CPU CPU CPU

CPU CPU CPU

I/O

I/O

Mem Mem Mem

I/O

I/O

Mem Mem Mem

11

RAC on OVM Disk Configuration (Production)

# # xen xen config config file file example example for for RAC RAC Guest Guest Domain Domain name name = = vmrac1" vmrac1" memory memory = = 8192" 8192" disk disk = =[ [ 'file:/OVS/running_pool/6243sm/System.img,xvda,w', 'file:/OVS/running_pool/6243sm/System.img,xvda,w', 'file:/OVS/running_pool/6243sm/RAC_x86-xvdb.img,xvdb,w', 'file:/OVS/running_pool/6243sm/RAC_x86-xvdb.img,xvdb,w', 'phy:/dev/mapper/mpath3p1,xvdc,w!', 'phy:/dev/mapper/mpath3p1,xvdc,w!', 'phy:/dev/mapper/mpath4p1,xvdd,w!', 'phy:/dev/mapper/mpath4p1,xvdd,w!', 'phy:/dev/mapper/mpath5p1,xvde,w!', 'phy:/dev/mapper/mpath5p1,xvde,w!', 'phy:/dev/mapper/mpath6p1,xvdf,w!', 'phy:/dev/mapper/mpath6p1,xvdf,w!', phy w! 'phy:/dev/mapper/mpath7p1,xvdg,w!', 'phy:/dev/mapper/mpath7p1,xvdg,w!', ] ] vif vif = =[ [ 'mac=00:16:3E:00:00:08, 'mac=00:16:3E:00:00:08, bridge=xenbr0', bridge=xenbr0', type=netfront, type=netfront, 'mac=00:16:3E:10:A5:96, 'mac=00:16:3E:10:A5:96, bridge=xenbr1', bridge=xenbr1', type=netfront, type=netfront, ] ]

phy

W!

12

RAC on OVM Network Configuration

O r a c le V M
G uest 1
P u b lic n e tw o rk P r iv a te n e tw o r k P u b lic n e tw o rk

G uest 2
P r iv a t e n e tw o rk

e th 0

e th 1

e th 0

e th 1

D o m a in -0

x e n b r0

xen b r1

bond0

bond1

e th 0

e th 1

e th 2

e th 3

P u b lic N e tw o rk S w itc h 1

P u b lic N e tw o r k S w itc h 2

P riv a te N e tw o r k S w itc h 1

P riv a te N e tw o rk S w itc h 2

13

Oracle RAC on Oracle VM

Non-Production deployment

What do I need to try this myself ?

Service Listener Service Listener instance 1 Oracle ASM
Oracle Clusterware

2-node Test RAC Minimum Requirements Two or more cores 4GB of memory or more 30GB of disk or more
Dom0 OS DOM 0

instance 2 Oracle ASM

Oracle Clusterware

GuestSystem OS Operating Guest VM

GuestSystem OS Operating Guest VM

DOM-0

Hypervisor DM-0
X86/64 Bare-Metal Server
CPU CPU CPU

I/O

I/O

Mem Mem Mem

14

Simple 2 Node Interview

15

Building a Cluster After initial boot, to build a cluster, simply run:

/u01/racovm/buildcluster.sh Automatically builds a cluster based on netconfig.ini & params.ini Automatically writes a build log and progress log

Internal testers reported:

2 nodes, 18 minutes 4 nodes, 23 minutes End-to-end, including DBCA

16

NEW: DeployCluster Tool Features Allows for fully automated end-to-end cluster deployment of Nnode clusters
Assuming VMs are pre-created w/NICs & shared disks

No Dom0 access or login to VMs is needed All previously released templates are fully compatible
As long as the OVMAPI enabled OS disk is used

Works on Oracle VM 3.0 and 3.1 Option to simulate a deployment attempt

Use the --dryrun (-D) flag

Allows sending any random key(s) to VMs

Use the --extrakeys (-K) flag

Tool options available in: deploycluster.ini

17

NEW: DeployCluster Tool Features (Contd) Allow ANY build configuration (SID name, user name, passwords, ports, etc.) to be modified from outside the guests at deploy time
Supply a custom params.ini using --params (-P) flag

VMs can be identified by UUID or simple name

Simple name may include the "*" and "?" wildcard characters Duplicate or illegal names detected on actual command line or on Oracle VM Manager

All exceptions are trapped showing a default of 4 lines from it Automated logfile written upon each invocation Easy to re-attempt a failed deployment
Only fix what failed, and leave other VMs running

18

DeployCluster Oracle VM Login Options

-u <username>, --username=<username> Username to connect to Oracle VM Manager -p <password>, --password=<password> Password to connect to Oracle VM Manager -H <host>, --host=<host> Manager hostname (use either -H or -U or none) -U <url>, --url=<url> Login URL to Manager (default: tcp://localhost:54321 or tcps://host:54322 when -H used to remote node) To configure secure connection to Manager, see Note#1456338.1
https://support.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=1456338.1

19

DeployCluster RAC OVM Template Options

-L, --list_vms_only List VMs seen via Oracle VM Manager; Honors -M flag -M <List of VMs>, --vms=<List of VMs> List of existing VM names or IDs to deploy cluster on. Supports "*" & "?" wildcard characters -P <params.ini>, --params=<params.ini> Location of params.ini file (sent to VMs) -N <netconfig.ini>, --netconfig=<netconfig.ini> Location of netconfig.ini file (sent to VMs) -B <yes|no>, --buildcluster=<yes|no> Start a buildcluster post-network setup (default: yes. [If netconfig_args passed then default: no]) -G <args>, --netconfig_args=<args> Advanced: Arguments to netconfig; override defaults -K <zip file>, --kitfile=<zip file> Advanced: Unzip new (partial) kitfile inside the VMs -X <file>, --extrakeys=<file> Advanced: File containing extra keys to send all VMs -D, --dryrun Show what will be done (do not start VMs or send msgs)
20

DeployCluster Examples List all VMs with a simple name of racnode.? on Manager
$ deploycluster.py u admin N netconf.ini M racnode.? -L

Deploy all VMs with a simple name of racnode.?:

$ deploycluster.py u admin N netconf.ini M racnode.?

See output on following slide

Deploy a VM as the 3rd node in an existing cluster

$ deploycluster.py u admin N netconf.ini M racnode3 G n3

21

Oracle RAC OneCommand (v1.1.0) for Oracle VM - deploy cluster - (c) 2011-2012 Oracle Corporation (com: 26700:v1.1.0, lib: 126247:v1.1.0, var: 1100:v1.1.0) - v2.4.3 server1.us.oracle.com (x86_64) Invoked as root at Sun Jun 3 23:28:53 2012 (size: 37500, mtime: Wed May 16 00:13:19 2012) Using: ./deploycluster.py -u admin -p **** -M racnode.? -P params-sample.ini -N netconfig30-3nodes.ini INFO: Attempting to connect to Oracle VM Manager... INFO: Oracle VM Client (3.1.1.305) protocol (1.8) CONNECTED (tcp) to Oracle VM Manager (3.1.1.212) protocol (1.8) IP (141.22.242.50) UUID (0004fb0000010000f2b5b95576e2c301) INFO: Inspecting /home/oracle/netconfig30-3nodes.ini for number of nodes defined.... INFO: Detected 3 nodes in: /home/oracle/netconfig30-3nodes.ini INFO: Located a total of (3) VMs; 3 VMs with a simple name of: ['racnode.0', 'racnode.2', 'racnode.1'] INFO: Verifying all (3) VMs are in Running state INFO: VM with a simple name of "racnode.0" is in a Stopped state, attempting to start it...OK. INFO: VM with a simple name of "racnode.2" is in a Stopped state, attempting to start it...OK. INFO: VM with a simple name of "racnode.1" is in a Stopped state, attempting to start it...OK. INFO: Detected that all (3) VMs specified on command have (3) common shared disks between them (ASM_MIN_DISKS=3) INFO: The (3) VMs passed basic sanity checks and in Running state, sending cluster details as follows: netconfig.ini (Network setup): /home/oracle/netconfig30-3nodes.ini params.ini (Overall build options): /home/oracle/params-sample.ini buildcluster: yes INFO: Starting to send cluster details to all (3) VM(s)...... INFO: Sending to VM with a simple name of "racnode.0".............. INFO: Sending to VM with a simple name of "racnode.2"..... INFO: Sending to VM with a simple name of "racnode.1"...... INFO: Cluster details sent to (3) VMs... Check log (default location /u01/racovm/buildcluster.log) on build VM (racnode.0)... INFO: deploycluster.py completed successfully at 23:30:26 in 92.6 seconds (01m:32s) Logfile at: /home/oracle/deploycluster/deploycluster1.log

22

Configuration parameters netconfig.ini (Network information)

Node name/IP, (6 items per node) NIC information Cluster name Mostly fixed data

params.ini (Build related options)

Database name, user names, disks, etc. More dynamic data

23

Building Clusters with more than 2 nodes

True Silent Install

Customer supplies an initialisation file. (netconfig.ini) Stamp file into shared storage Repeat this section, identifying the 6 attributes for each node Power on new nodes and pass command on grub boot-up line

netconfig.ini
# Node specific information NODE1=test170 NODE1IP=192.168.1.170 NODE1PRIV=test170-priv NODE1PRIVIP=10.10.10.170 NODE1VIP=test170-vip NODE1VIPIP=192.168.1.172

NODE2=test171 NODE2IP=192.168.1.171 NODE2PRIV=test171-priv NODE2PRIVIP=10.10.10.171 NODE2VIP=test171-vip NODE2VIPIP=192.168.1.173

24

OVM2: Building a Cluster from DOM-0 Stamp netconfig.ini to shared disk:

# ./netconfig.sh -W -c /dev/sdxyz

If filesystem disk, use losetup vf to loop mount the disk, then stamp the loop device.

Boot VMs with node number hint (on Dom0):

# xm create -c vm.cfg extra="console=hvc0 template-reconfig-args=-R -n1 -b" (node 1) # xm create -c vm.cfg extra="console=hvc0 template-reconfig-args=-R -n2" (node 2) -R read network config -n1 node 1 -b build node

Above will automatically configure the network on the new VMs and build a 2 node cluster
25

Running RACOVM directly List all steps:

/u01/racovm/racovm.sh -l

Running individual steps in sequence:

/u01/racovm/racovm.sh -S setsshroot,checklocal

Most steps can be run global or local (add local to step) Any failure of any step will stop execution Combination of common steps are also available as special steps, e.g. buildcluster or command line flags, e.g. -c To cleanup run:
/u01/racovm/racovm.sh -S clean

26

Running RACOVM directly (Contd) Each invocation prints this banner:

Invoking on test125 as root... Oracle RAC 11gR2 OneCommand (v1.1) for Oracle VM - (c) 2010 Oracle Cksum: [4028880687 234100 racovm.sh] at Mon Oct 4 14:14:14 EDT 2010

Kernel: 2.6.18-164.0.0.0.1.el5xen (i686) [2 processor(s)] 1800 MB Step(s): setsshroot,checklocal

Timing for each operation:

2010-10-04 14:48:16:[buildcluster:Start:test235] Build 11gR2 RAC Cluster 2010-10-04 15:32:36:[buildcluster:Done :test235] Build 11gR2 RAC Cluster 2010-10-04 15:32:36:[buildcluster:Time :test235] Completed successfully in 2660 seconds (0h:44m:20s)

27

RACOVM Command Line Options

Usage: racovm.sh (v1.3) RAC OVM main driver Oracle RAC 11gR2 OneCommand (v1.3) for Oracle VM - (c) 2010-2011 Oracle -c : Builds a new RAC cluster including database [buildcluster] -g : Configures & starts Grid Infrastructure on all nodes calls: setsshroot,copykit,usrsgrps,printparams,setsshora,diskconfig,check, creategrid,cvupostcrs,clusterstate -r : Configures RAC Home by running clone.pl on all nodes (no DBCA) [racclone] -d : Run DBCA to create the RAC database [createdb] -p : Setup passwordless ssh for Oracle users between all nodes [setsshora] -P : Setup passwordless ssh for root user between all nodes [setsshroot] -S <stepname>,<stepname> ... : Run individual steps (comma separated list) stops execution on first error -N <node3>,<node4> .. : List of nodes to add or remove Required for 'addnodes' 'removenodes' 'addinstances' 'deleteinstances' Optional for 'createdb' -l : List all available steps -s : Skip confirmations -h : Prints this help message NOTE: Specify one of the above options in each invocation (-S flags allows multiple steps). The -c flag is equivalent to -g, -r, -d called in sequence. See params.ini configuration file for further options

28

Adding or Removing Node(s) / Instance(s) Fully automated addition and removal of nodes or instances Simply run:
./racovm.sh -S addnodes -N node2,node3 Or: ./racovm.sh -S removenodes -N node2,node3 Or: ./racovm.sh -S addinstances -N node2,node3 Or: ./racovm.sh -S deleteinstances -N node2,node3

29

diskconfig.sh Configures disks in VMs Verifies disks are not held on any node by
ASM, ASMLib, RAID device, PowerPath, Device Mapper, User Application, Filesystem, Swap Device

Stamps and discovers disks on all nodes (verify sharedness) Auto-partition & align data to 1MB offset (default) Supports MSDOS or GPT partition table Merges needed udev rules to /etc/udev/
Supports EL4, EL5 & SLES10, SLES11

Supports but does not do the initial creation of

ASMLib Multipath

30

netconfig.sh Configures network in VMs Full validation on user input, NIC names, IP/subnet masks Checks for duplicate IPs on subnet (arping) Writes /etc/hosts and related ifcfg-*, resolv.conf, etc. files to fully configure network Allows stamping of netconfig.ini to shared storage; helps in N-node network configuration (from dom0 or inside guests) Supports and configures bonding (not needed inside guests)

31

doall.sh Run command on all nodes

doall.sh [options] <command> | "<command1>;<command2>;..<commandX>" -l <user> : Login as user (default: current user) -x : Run commands in sequence (default: run in parallel) -P : Print commands as they are executed (default: noprint) -p : Setup passwordless SSH for <user> on all nodes listed in netconfig.ini -s : Skip confirmation for passwordless SSH setup -u : Skip unreachable nodes -U : Skip nodes that are unreachable or without passwordless SSH configured -L first|last|skip : Local node execution order -N <node1>,<node2> .. : Nodes to run on (default: all nodes in netconfig.ini)

Examples:
./doall.sh -L last reboot ./doall.sh -ps /u01/app/11.2.0/grid/bin/diagcollection.sh

32

Useful Links Oracle RAC OVM Templates download locations

From e-delivery: 11.1.0.7.* & 11.2.0.1.* 11.1:http://www.oracle.com/technetwork/database/clustering/overvi ew/rac-092962.html 11.2:http://www.oracle.com/technetwork/database/clustering/overvi ew/rac-template-11grel2-166623.html From My Support http://support.oracle.com/ (support subscription): 11.2.0.2.0 Patch# 10113572 11.2.0.2.2 Patch# 11836831 11.2.0.3.0 Patch# 12694014 11.2.0.2.6 Patch# 14000016 11.2.0.3.2 Patch# 13818831 All described in Note:1185244.1:
https://support.oracle.com/oip/faces/secure/km/DocumentDisplay.jspx?id=1185244.1

Training Video RAC SIG (9-Dec-2010, 8-DEC-2011)

http://www.oracleracsig.org/pls/apex/f?p=105:206:::NO

33

QUESTIONS ANSWERS

35

36