Split Mirror Backup and

Database Replication Using EMC

TimeFinder

Enhancing Oracle Applications Backup

Requirements with EMC Enterprise Storage
Solutions
Prepared by Oracle IFAD Performance Group and EMC Enterprise Alliances

Authors: Shankar P Kodumudi, Oracle IFAD Performance Group

Paul Manning, EMC Enterprise Alliances
Last Updated: February 8, 1999
Version: 1.0

Copyright 1999 All Rights Reserved

Oracle Corporation and EMC Corporation
 Contents

Executive Summary....................................................................................................................... 3

Introduction.................................................................................................................................... 4

EMC Split Mirroring Technology ................................................................................................ 5

Configuration ................................................................................................................................. 6

Specific Tests .................................................................................................................................. 7

Summary......................................................................................................................................... 8

Appendix A - Replicating Oracle Application Database.......................................................... 9

Appendix B - Replicating Oracle Applications Code ............................................................. 12

Preface
This white paper reflects testing that was done to validate the use of EMC technology in
conjunction with Oracle Applications. Intended to be a checklist for the DBA group
experienced with Oracle Applications installations; certain instructions are specific to
each application environment.

EMC2 , EMC, and Symmetrix and registered trademarks and EDM, SRDF, TimeFinder,
EMC Enterprise Storage, and The Enterprise Storage Company are trademarks of EMC
Corporation. Other trademarks are the property of their respective owners.

This paper is being distributed by EMC Corporation for information purposes only. EMC
Corporation does not warrant that this document is free from errors. No contract is
implied or allowed.

Copyright 1999. All rights reserved.

G824 4/99
Executive Summary
The testing of the split mirror technology in conjunction with Oracle Applications was
done to validate the use of EMC TimeFinderTM and to document best practices for the use
of this combination. Although the testing was done using customer data on a specific
platform, the methods and results are extensible to many Oracle Applications Modules as
well as many operating system platforms. In this specific set of tests, we used the HP
platform and the Oracle Applications GEMS module to validate our results. Tests for
both the cold backup and hot backup method were completed. We also used TimeFinder
to present the split mirror (third mirror) copy of our application and database to the same
host/CPU, which ran the original copy of our application and database. In doing this
testing, we highlighted the capabilities, which enable an Oracle Applications
solution to be available for a greater percentage of the 7x24 time frame with no impact on
its performance, while its backup was being done.

3
Introduction
The need for concurrent copies of companies application databases is increasing as the
demand for worldwide access grows. The cost of both scheduled and unscheduled
downtime for a production application has become a business-critical concern. As the
new technology of split mirrors provides a means to reduce downtime and provide a
point-in-time copy of the application and database, many Oracle and EMC customers
want to make sure this works for their specific environment. As such, EMC and Oracle
have established a 12 GB customer database on which the GEMS application was run.
The use of the EMC TimeFinder product provided a detachable copy of the database and
application, which could be run concurrently with the production database.

The purpose of our testing was to validate the use of TimeFinder with both the cold and
hot backup methods of Oracle Applications running on a live Oracle database and to
document the process (providing best practice guidelines).

4
EMC Split Mirroring Technology
EMC is the leading independent storage hardware and software provider. In addition to
the Symmetrix 3330 hardware, we also leveraged the EMC TimeFinder product.
TimeFinder is a software product that allows a detachable copy of a mirror device. This
is often referred to as a third mirror copy because in a standard RAID 1 environment, we
already have two mirrored pairs for each device the host systems reads and writes to.
The TimeFinder third mirror is referred to as a Business Continuance Volume (BCV)
which is managed by the Symmetrix storage cache (independent from the host CPU).
This enables greater performance and flexibility than other solutions found in the market
today will offer.

Two important features, which set the EMC technology apart from other third mirror
technologies available today, are 1) The mirroring is managed by the cache within the
Symmetrix and is not taking valuable cycles away from the host system. This has a big
performance gain on a system, which is already under load. 2) The re-synchronization
process, as well as the restore, have the ability to do an incremental versus a full re-
silvering which can save a significant amount of time as well as compute cycles.

The TimeFinder command line interface (CLI) enables the control of the detachable
mirror copy. They can be associated to a given device, re-silvered and split. One very
important feature of TimeFinder is that once a BCV has been split from the primary
device, it can be incrementally re-synchronized. This means that only those tracks which
have been changed on the primary or BCV will need to be copied, which saves a
significant amount of time in a large database environment.

EMC products and terminology are described on the EMC Web site at
http://www.EMC.com.

Oracle
Applications

Oracle
Database

Primary Copy
Protected with
RAID 1

Business
Continuance
HP K570 Volume (BCV)

EMC Symmetrix

5
Configuration
The hardware configuration used in this testing includes an HP K570 connected to an
EMC Symmetrix 3330. The software used was HP-UX 10.20, EMC TimeFinder,
Oracle Applications, and a sample database. The HP Logical Volume Manager (LVM)
was used to manage the volumes and EMC TimeFinder command line interface (CLI)
was used to manage the third mirror synchronization of the BCVs.

Diagram 1. Hardware and Software Configuration

HP K570
Oracle Apps Copy of Oracle Apps Copy of Oracle Apps
Oracle Apps Code
Database Code Database

Filesystem /u05 /u06 /u05bcv /u06bcv

Logical
Volume vg05 vg06 /u06bcv v06bcv
Groups

c4t8d2

c4t8d0

d1

d2

Symmetrix 3330
6
Specific Tests and Best Practices
In the Oracle Applications environment, it is essential to have regular backups taken to
ensure recovery from any possible failures. Most customers will need to either plan
downtime or be able to run production in a state of reduced performance while they are
taking a backup of their environment. The amount of time it takes to perform the backup
will depend on the size of the database, the number of tape drives and the amount of
parallel streams which can be engaged to back up the database. In the case of the hot
backup, the amount of activity running on the production database can also add delay to
the process.

With EMC TimeFinder, operations managers can shorten or eliminate the amount of
downtime as well as the amount of time spent running in the reduced performance
mode. This capability can be further enhanced by means of leveraging the extremely
large amount of cache the Symmetrix storage system provides.

We configured the systems to provide both a primary mirror (RAID 1) and a third mirror
(BCV) for each of the four volumes on which our database and application resided. This
requires a good mix of skills from Oracle DBA, EMC mirror management and HP logical
volume management. In many large production environments, it is critical that these
three skill sets all understand the full process conceptually as well as tactically in that
failure to understand how their specific part effects the entire process can be a cause of
much confusion.

Once the production database and application were up and running, the test was
conducted on both a live database as well as a cold database. In the case of the hot
backup, we had many transactions in fight at the time we split the mirrors.

Appendix A and B provide detailed steps used to do both the cold and hot backup.

7
Summary
The results of our testing confirmed that the combination of EMC TimeFinder with
Oracle Applications works, as we would expect it to. The use of this solution to greatly
enhance the speed and flexibility of backing up a production Oracle Applications and
database is a reliable solution. The process involves a good knowledge of system
administration (LVM), database management (DBA), and EMC third mirroring
technology and that all parties involved in supporting this solution see their role in the
overall solution.

8
Appendix A - Replicating Oracle Application Database
1. Prepare the new location by setting up the directory structure and location where the
datafiles will reside as well as the admin directory.

e.g.
as the Root OS user:
Vgchgid /dev/rdsk/c4t8d0 /dev/rdsk/c4t8d1 /dev/rdsk/c4t8d3
mkdir /dev/vg06bcv
mknod /dev/vg06bcv/group c 64 0X120000
vgimport /dev/vg06bcv /dev/dsk/c4t8d0 /dev/rdsk/c4t8d1
/dev/rdsk/c4t8d3
Vgchange -a y /dev/vg06bcv
vgcfgbackup /dev/vg06bcv
mkdir /u06bcv
mount /dev/vg06bcv/lvol1 /u06bcv

2. Rename the files under the pfile directory to match the NEWSID name.

e.g.
mv initSID.ora initNEWSID.ora
mv initSID_0.ora initNEWSID_0.ora

3. Make links from $ORACLE_HOME/dbs to the location of initNEWSID.ora and

initNEWSID_0.ora.

e.g.
cd $ORACLE_HOME/dbs
ln -s <$ORACLE_BASE>/admin/NEWSID/pfile/initNEWSID.ora initNEWSID.ora
ln -s <$ORACLE_BASE>/admin/NEWSID/pfile/initNEWSID_0.ora
initNEWSID_0.ora

4. Edit each of the files under the pfile directory. Ensure that the SID has been changed
to NEWSID in the DB_NAME parameter, the control file and output dump
directories have been modified to reflect the new location of these files, and the other
directory paths such as ifile are correct.

5. Add new SID information to tnsnames.ora, listener.ora, and oratab.

e.g.
tnsnames.ora:
NEWSID = (DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)(HOST=<host_name>)(PORT=1521))
(CONNECT_DATA=(SID=NEWSID)))
listener.ora:
add (ADDRESS = (PROTOCOL = ipc)(KEY = NEWSID)) to address_list and
add (SID_DESC=(SID_NAME=NEWSID)(ORACLE_HOME =
/u01/dba/oracle/product/7.1.6)) to sid_list
oratab:
add entry NEWSID:/u01/dba/oracle/product/7.1.6:Y

9
6. With the current database up, issue a create CREATE CONTROLFILE command.

e.g.
sqldba lmode=y
SQLDBA> connect internal
SQLDBA> alter database backup control file to trace;

It will create a trace file in your trace directory (udump).

7. Copy this trace file to your $ORACLE_BASE/admin/NEWSID/create directory and

call it something like ccf.sql.

8. Edit ccf.sql by removing everything except the CREATE CONTROLFILE comand.

9. Edit the CREATE CONTROLFILE command by changing the word REUSE to SET,
and NORESETLOGS to RESETLOGS, and modify the dbname. Make any
modifications to the file names to reflect their new location.

10. Shutdown the current database.

11. Ensure that your environment is set up with the new SID for ORACLE_SID.

12. Run the create controlfile command.

e.g.
cd <location where ccf.sql is located from step 6>
sqldba lmode=y
SQLDBA> connect internal
SQLDBA> startup nomount
SQLDBA> @ccf.sql
SQLDBA> alter database open resetlogs;
SQLDBA> shutdown immediate;
SQLDBA> startup

13. The current database can be brought back up at this time.

16. Verify/Create database link in the new setup:

sqlplus system/manager;
SQL> drop public database link ofi_to_ess;
SQL> create public database link ofi_to_ess connect to ess_int identified by ess_int
using ESS NEWSID;
SQL> commit;

17. Change system and sys oracle passwd and make entry in /etc/orapwd (make sure
this file is readable only by root).

10
18. Ensure the /etc/tnsname.ora entry is correct by testing the db link (this also tests the
validity of the db link).

Example:

Logon to GEMMS/OF database via sqlplus and type:

SQL> desc user_tables@ofi_to_ess

The user_tables on the ESS database should be displayed here.

19. Ensure the /etc/listener.ora entry is correct and is effective. You can do this by
reloading the listener so that current SQLNet connections are not disconnected. For
example:

Set your Oracle environment (via iadenv script)

$ lsnrctl reload [listener_name]

$ lsnrctl status [listener_name]

11
Appendix B - Replicating Oracle Applications Code
1. Create a new APPL_TOP location to place the new application code if necessary.
Some replicated instances will share application code. OFA standards should be
used when creating the directory path.

e.g.
Vgchgid /dev/rdsk/c4t8d2
mkdir /dev/vg05bcv
mknod /dev/vg05bcv/group c 64 0X110000
vgimport /dev/vg05bcv /dev/dsk/c4t8d2.
Vgchange -a y /dev/vg05bcv
vgcfgbackup /dev/vg05bcv
mkdir /u05bcv
mount /dev/vg05bcv/lvol1 /u05bcv

2. Create new OS user applNEWSID to be used to start and stop the concurrent
managers.

Modify the .profile for applNEWSID to set environment variables and other
miscellaneous variables that are set for typical users.

e.g.
umask 022
ORACLE_SID=NEWSID
ORACLE_HOME=/u01/dba/oracle/product/7.1.6; export ORACLE_HOME
ORACLE_BASE=/u01/dba/oracle; export ORACLE_BASE
PATH=$PATH:$ORACLE_HOME/bin
PATH=$PATH:/bin:/usr/bin:/usr/ucb:.
PATH=$PATH:/usr/local/bin
PATH=$PATH:$ORACLE_BASE/local/scripts
export PATH
. $APPL_TOP/APPLSYS.env
stty erase "^?"
TERM=vt100

3. Add logNEWSID and outNEWSID under the common directory.

e.g.
cd /u02/applmgr/common
mkdir logNEWSID outNEWSID

4. Once the code is copied, remove all the old log files and out files.

e.g.
cd $FND_TOP (ie /u02/applmgr/10.6_NEWSID/fnd/*)
rm -r log* out*
mkdir log out logNEWSID outNEWSID

5. Modify $APPL_TOP/APPLSYS.env and CONCSUB_profile by changing the

$APPL_TOP to be the correct location and changing all SID to NEWSID.

12