Installing Oracle RAC 11g Release 2 (11.2.0.

2) in Storage Foundation for Oracle RAC environment

Starting with Oracle RAC 11.2.0.2, Oracle has introduced Redundant Interconnect Usage feature which needs multicast to be enabled on all private links. Currently if PrivNIC/MultiPrivNIC is configured to manage the cluster interconnect private IP addresses, when a private link fails, the PrivNIC/MultiPrivNIC agent fails over the IP addresses configured on the failed link to the available link. As a result, the ora.cluster_interconnect.haip resource becomes offline, Oracle cluster synchronization services daemon fails and restarts, but does not join the cluster till the Oracle grid infrastructure is restarted on all nodes. Symantec has raised an Oracle bug #10211330. We are working with Oracle to get this situation rectified and add support for co-existence of MultiprivNIC/PrivNIC with 11.2.0.2. In the interim, to make use of redundant networks for the interconnect, Oracle recommends that customers to use link aggregation/ NIC bonding and/or Redundant Interconnect Usage. This document provides the procedure for installing Oracle RAC 11gR2 (11.2.0.2) on a new SFRAC setup as well as procedure for upgrading from existing Oracle RAC 11gR2 (11.2.0.1) on a SFRAC setup. Refer to Veritas Storage Foundation for Oracle RAC and Veritas Storage Foundation Cluster File System for Oracle RAC Support Matrix for the SFRAC versions certified with Oracle RAC 11gR2 (11.2.0.2). See the Technical Support TechNote: http://www.symantec.com/business/support/index?page=content&id=TECH44807.

Fresh installation of Oracle RAC 11gR2 (11.2.0.2) on Linux and Unix

Configuring LLT to use link aggregation/ NIC bonding If you are planning to use link aggregation/ NIC bonding, make sure that a link configured under a aggregated link or NIC bond is not configured as a separate LLT link. When LLT is configured over a bonded interface, do one of the following steps to prevent GAB from reporting jeopardy membership: Configure an additional NIC under LLT in addition to the bonded NIC. Add the following line in the /etc/llttab file:
set-dbg-minlinks 2

Configuring Private IPs for Oracle cluster interconnect

If you are planning to use link aggregation/ NIC bonding configure the static private IP addresses for the bonded interface to be used as cluster interconnect, refer to Operating System documentation for configuring bonded NIC and the static IP addresses for an interface. If you are planning to use Redundant Interconnect Usage feature, configure the static private IP addresses for the private interfaces to be used as cluster interconnect. It is recommended to use all the LLT links as cluster interconnects. Refer to Operating System documentation for procedure to configure the static IP addresses for an interface. Oracle needs multicast to be enabled on the private interconnect for Redundant Interconnect Usage feature. Refer to Oracle document ID 1212703.1 to verify multicast requirements are met.

Installing Oracle RAC

Refer to the procedure provided in Installing Oracle RAC and Performing Oracle RAC post-installation tasks in Veritas Storage Foundation for Oracle RAC Installation and Configuration Guide for installing Oracle RAC 11gR2 (11.2.0.2) in SFRAC setup, with following changes. During the Grid Infrastructure installation, identify all the LLT links as private. In the section Completing Oracle RAC post-installation tasks, configuring the CSSD resource under VCS control using the SF Oracle RAC Installer will fail since PrivNIC/MultiPrivNIC resource is not configured. Hence, you would require to configure cssd resource under VCS control manually. While configuring cssd resource manually, do not set the dependency of the CSSD resource on the PrivNIC or MultiPrivNIC resources.

Configuring LLT links(Private Interconnect links) in the GPnP profile

If you are planning to use Redundant Interconnect Usage feature, it is recommended to use all the LLT links as cluster interconnects. If any of the LLT interfaces is not configured as cluster interconnect in GPnP profile, update the GPnP profile to include the remaining LLT links, which were not added to the profile during the Oracle Grid Infrastructure installation. To configure the LLT links in the GPnP profile 1. View the currently configured interfaces:
# $GRID_HOME/bin/oifcfg getif eth0 10.2.156.0 global public eth1 192.168.12.0 global cluster_interconnect

The interfaces that are currently stored in the GPnP profile, their subnets, and their role (public or cluster_interconnect) are displayed. 2. Add the remaining LLT links to the GPnP profile:
# $GRID_HOME/bin/oifcfg setif -global \ eth2/192.168.12.0:cluster_interconnect

3. Verify that the correct interface subnet is in use:

# $GRID_HOME/bin/oifcfg getif eth0 10.2.156.0 global public eth1 192.168.12.0 global cluster_interconnect eth2 192.168.2.0 global cluster_interconnect

Upgrading to Oracle RAC 11gR2 (11.2.0.2) from Oracle RAC 11gR2 (11.2.0.1) on Unix
To upgrade existing 11.2.0.1 Oracle Grid Infrastructure installations to Oracle Grid Infrastructure 11.2.0.2, you must do either of the following: i. Patch the Oracle Grid Infrastructure home with 9413827 and 9706490 patches. (Solaris) ii. Install Oracle Grid Infrastructure Patch Set 1 (GI PSU1) or Oracle Grid Infrastructure Patch Set 2 (GI PSU2). Oracle Clusterware and Oracle ASM upgrades are out-of-place upgrades.

To upgrade to Oracle RAC 11gR2 (11.2.0.2)

1. Freeze cvm service group and delete PrivNIC/MultiPrivNIC agents, if any. If Oracle database is under VCS control, freeze database service group.
# # # # # haconf makerw hagrp -freeze cvm hagrp freeze <database service group> hares delete <PrivNIC/MultiPrivNIC resource> haconf -dump makero

2.

Configure the static private IP addresses for the private interfaces used as cluster interconnect as detailed in the section Configuring Private IPs for Oracle cluster interconnect. Earlier these private IP addresses were managed by PrivNIC/MultiPrivNIC. Install the prerequisite Oracle patches for upgrading to 11.2.0.2 as listed above in Oracle Grid Infrastructure home. Refer to the patch READMEs for instructions. For upgrading Oracle grid infrastructure, Refer to How to Upgrade to Oracle Grid Infrastructure 11g Release 2 section in Oracle Grid Infrastructure Installation Guide 11g Release 2 (11.2) for more details. Log in as the Oracle grid user. On the first node, set the DISPLAY variable.

3.

4.

For Bourne Shell (bash), type:

$ DISPLAY=10.20.12.150:0.0; export DISPLAY

where 10.20.12.150 is the IP address of X client where you want to export the display for the installer. For C Shell (csh or tcsh), type:
$ setenv DISPLAY 10.20.12.150:0.0

where 10.20.12.150 is the IP address of X client where you want to export the display for the installer. 5. Start the Oracle Universal Installer for installing Oracle grid infrastructure, and select the option to upgrade an existing Oracle Clusterware and Oracle ASM installation.
$ <11.2.0.2 installer directory>/runInstaller

Note: Provide a new grid infrastructure home when the OUI prompts for the grid infrastructure home location. Oracle Clusterware and Oracle ASM upgrades are out-of-place upgrades. 6. After completing the grid infrastructure installation, when prompted run rootupgrade.sh script as superuser on each node. Note: Veritas Membership library would be automatically linked with grid infrastructure during installation. For upgrading the Oracle database, refer to Oracle Database Upgrade Guide for complete details. 7. Log in as the Oracle database user. On the first node, set the DISPLAY variable.

For Bourne Shell (bash), type:

$ DISPLAY=10.20.12.150:0.0;export DISPLAY

where 10.20.12.150 is the IP address of X client where you want to export the display for the installer. For C Shell (csh or tcsh), type:
$ setenv DISPLAY 10.20.12.150:0.0

where 10.20.12.150 is the IP address of X client where you want to export the display for the installer. 8. 9. Start the Oracle Universal Installer for installing database software on the first node.
$ <11.2.0.2 installer directory>/runInstaller

After upgrading the Oracle database, relink Veritas ODM library. Refer to the procedure provided in Relinking the SF Oracle RAC libraries with Oracle RAC in Veritas Storage Foundation for Oracle RAC Installation and Configuration Guide. When prompted for Oracle home, provide the Oracle 11.2.0.2 database home.
$ORACLE_HOME/bin/dbua

10. Use the database upgrade assistant to upgrade the existing databases. As a Oracle database user, enter: 11. If DB is under VCS control , then configure the Oracle RAC database for manual startup, refer to section Preventing automatic database startup in Veritas Storage Foundation for Oracle RAC Installation and Configuration Guide. 12. Unfreeze cvm service group and if database is under VCS control then unfreeze database service group.
# # # # haconf makerw hagrp -unfreeze cvm hagrp unfreeze <database service group> haconf -dump makero

Upgrading to Oracle RAC 11gR2 (11.2.0.2) from Oracle RAC 11gR2 (11.2.0.1) on Linux
The following procedure assumes that you are running Oracle RAC 11gR2 (11.2.0.1) with Oracle patch 8649805. Upgrading to Oracle RAC 11gR2 (11.2.0.2) requires Oracle Patch Set Update 11.2.0.1.2 (PSU2) released for Oracle RAC 11gR2 (11.2.0.1). Apply it on all the cluster nodes.

To apply Oracle Patch Set Update 11.2.0.1.2 (PSU2)

13. Freeze cvm service group and disable PrivNIC/MultiPrivNIC agents, if any. If database is under VCS control, freeze the database service group
# # # # # haconf makerw hagrp -freeze cvm hagrp freeze <database service group> hares delete <PrivNIC/MultiPrivNIC resource> haconf -dump makero

14. Configure the static private IP addresses for the private interfaces used as cluster interconnect as detailed in the section Configuring Private IPs for Oracle cluster interconnect. Earlier these private IP addresses were managed by PrivNIC/MultiPrivNIC. 15. Stop the database. Run the following command on one of the cluster nodes as the Oracle database user:
$ srvctl stop database -d <database_name>

16. Stop Oracle Grid Infrastructure on each node in the cluster. As a superuser, enter:
# crsctl stop crs

17. Unlink Veritas Membership library. As a Oracle grid user, on each node enter:
$ cd $GRID_HOME/lib $ mv libskgxn2.so libskgxn2.so.veritas.`date '+%m_%d_%y-%H_%M_%S'` $ cp libskgxns.so libskgxn2.so

18. Start Oracle Grid Infrastructure on each node in the cluster.

# crsctl start crs

19. Roll back Oracle patch 8649805 from Oracle grid infrastructure home. Refer to the Oracle patch 8649805 readme for rollback procedure.
20. Roll back Oracle patch 8649805 from Oracle database home. Refer to the Oracle patch 8649805 readme for rollback

procedure. 21. Upgrade the OPatch utility of Oracle grid Infrastructure home and Oracle database software home by applying Oracle patch 6880880. a. Extract the OPatch utility in the Oracle grid infrastructure home. As a Oracle grid user, on each node in cluster enter:
$ unzip <6880880 patch zip file> -d $GRID_HOME

b. Extract the OPatch utility in the Oracle database software home. As a Oracle database user, on each node in cluster enter:
$ unzip <6880880 patch zip file> -d $ORACLE_HOME

22. Apply Oracle Set Update 11.2.0.1.2 (PSU2) to Oracle grid infrastructure home. As a superuser, on each node in cluster enter:
# $GRID_HOME/OPatch/opatch auto <PSU2 patch file directory> -oh $GRID_HOME

Note: This applies Oracle patch 9655006 and 9654983. 23. Apply Oracle Set Update 11.2.0.1.2 (PSU2) to Oracle database home. As a Oracle database user, on each node in cluster enter:
$ORACLE_HOME/OPatch/opatch apply <PSU2 patch file directory>/9654983 oh $ORACLE_HOME

To upgrade to Oracle RAC 11gR2 (11.2.0.2)

For upgrading Oracle grid infrastructure, refer to How to Upgrade to Oracle Grid Infrastructure 11g Release 2 section in Oracle Grid Infrastructure Installation Guide 11g Release 2 (11.2) for more details. 24. Log in as the Oracle grid user. On the first node, set the DISPLAY variable.

For Bourne Shell (bash), type:

$ DISPLAY=10.20.12.150:0.0; export DISPLAY

where 10.20.12.150 is the IP address of X client where you want to export the display for the installer. For C Shell (csh or tcsh), type:
$ setenv DISPLAY 10.20.12.150:0.0

where 10.20.12.150 is the IP address of X client where you want to export the display for the installer. 25. Start the Oracle Universal Installer for installing Oracle grid infrastructure, and select the option to upgrade an existing Oracle Clusterware and Oracle ASM installation.

$ <11.2.0.2 installer directory>/runInstaller

Note: Provide a new grid infrastructure home when the OUI prompts for the grid infrastructure home location. Oracle Clusterware and Oracle ASM upgrades are out-of-place upgrades. 26. After completing the grid infrastructure installation, when prompted run rootupgrade.sh script as superuser on each node. Note: Veritas Membership library would be automatically linked with grid infrastructure during installation. For upgrading the Oracle database, refer to Oracle Database Upgrade Guide for complete details. 27. Log in as the Oracle database user. On the first node, set the DISPLAY variable. For Bourne Shell (bash), type:
$ DISPLAY=10.20.12.150:0.0;export DISPLAY

where 10.20.12.150 is the IP address of X client where you want to export the display for the installer. For C Shell (csh or tcsh), type:
$ setenv DISPLAY 10.20.12.150:0.0

where 10.20.12.150 is the IP address of X client where you want to export the display for the installer. 28. Start the Oracle Universal Installer for installing database software on the first node.
$ <11.2.0.2 installer directory>/runInstaller

29. After upgrading the Oracle database, relink Veritas ODM library. Refer to the procedure provided in Relinking the SF Oracle RAC libraries with Oracle RAC in Veritas Storage Foundation for Oracle RAC Installation and Configuration Guide. When prompted for Oracle home, provide the Oracle 11.2.0.2 database home. 30. 31. Use the database upgrade assistant to upgrade the existing databases. As a Oracle database user, enter:
$ $ORACLE_HOME/bin/dbua

32. If DB is under VCS control , then configure the Oracle RAC database for manual startup, refer to section Preventing automatic database startup in Veritas Storage Foundation for Oracle RAC Installation and Configuration Guide. 33. Unfreeze the cvm service group and if database is under VCS control, unfreeze database service group.
# # # # haconf makerw hagrp -unfreeze cvm hares unfreeze <database service group> haconf -dump makero