Business Continuity for Oracle E-Business Suite Release 12.

2 Using Oracle 12c

(12.1.0.2) Physical Standby Database (Doc ID 1963472.1)

Oracle E-Business Suite Release 12.2.x has numerous configuration options that can be chosen
to suit particular business scenarios, hardware capabilities, and availability requirements.

This document describes how to configure an Oracle E-Business Suite Release 12.2.x
environment to use Oracle Database 12c Release 1 (12.1.0.2) as a physical standby. This
document covers both Oracle RAC and non-RAC configurations.

The most current version of this note is available as My Oracle Support Knowledge Document
1963472.1, Business Continuity for Oracle E-Business Suite Release 12.2 Using Oracle 12c
(12.1.0.2) Physical Standby Database.

Note: While the general concepts in this paper apply to all operating systems and hardware
architectures that Oracle supports, the procedure has not been validated on the Windows
platform.

This document currently applies to symmetric primary and standby configuration only. There
are a few limitations with the configuration explained throughout this document. For example,
depending upon the configuration-1 explained in section-5, an ADOP cycle needs to be aborted
before reapplying a patch if a failover event occurs. If you use the configuration-2 as explained
in section-5, then an ADOP cycle can be resumed in case of a failover.

In This Document

This document is divided into the following sections:

Section 1: Overview
Section 2: Before You Start

Section 3: Preparing the Primary Database for Standby Database Creation

Section 4: Creating a Physical Standby Database

Section 5: Configuring the Application Tier After the Standby Database has been
Enabled
Section 6: Performing Role Transitions

Section 7: Maintaining Oracle E-Business Suite with a Standby Database

Appendix A: Oracle Net Files

 Appendix B: Using Oracle Applications Manager to Register a Standby Database

Appendix C: Example Standby Database Commands

Appendix D: Using RMAN to Create a Physical Standby Database

Appendix E: Using Data Guard Broker (DGMGRL) to Manage Standby Databases

Appendix F: Handling Online Patching Following a Standby Failover Event

Appendix G: Creating a Non-RAC Standby for a RAC Primary

There is a change log is at the end of this document.

A number of conventions are used in this document:

Convention Meaning
Machines running forms, web, concurrent processing and other servers.
Application tier
Sometimes called middle tier.
Database tier Machines running Oracle E-Business Suite database.
Primary Oracle E-Business Suite system, which will used to create a standby
Primary system
system.
Standby system Oracle E-Business Suite system created as a copy of the production system.
ORACLE
User account that owns the database file system (database ORACLE_HOME and
files).
CONTEXT_NAME
The CONTEXT_NAME variable specifies the name of the applications context that is
used by AutoConfig. The default is <SID>_<hostname>.
STNDBY_CONTEXT The default is <SID>_<stdby_hostname>.
Full path to the applications context file on the application tier or database tier.
The default locations are as follows:
CONTEXT_FILE
Application tier context file: <INST_TOP>/appl/admin/CONTEXT_NAME.xml

Database tier context file: <RDBMS_ORACLE_HOME>/appsutil/<CONTEXT_NAME>.xml

APPSpwd Oracle E-Business Suite database user password.
Monospace text
Represents command line text. Type such a command exactly as shown,
excluding prompts such as '%'.
Text enclosed in angle brackets represents a variable. Substitute a value for the
<>
variable text. Do not type the angle brackets.
On UNIX, the backslash character can be entered at the end of a command
\
line to indicate continuation of the command on the next line.
Primary TNS alias to connect to primary database.
database alias
Standby TNS
TNS alias to connect to standby database.
alias

Section 1: Overview

1.1 Standby Database

1.2 Oracle Data Guard
1.3 Oracle Data Guard Broker

1.1 Standby Database

A standby database is a transactionally consistent copy of the primary database. Using a backup
copy of the primary database, you can create up to 30 standby databases and incorporate them in
Oracle Data Guard configuration.

There are three types of standby databases:

Physical standby
Provides a physically identical copy of the primary database, with on-disk database
structures that are identical to the primary database on a block-for-block basis. The
database schema, including indexes, is the same. A physical standby database is kept
synchronized with the primary database by recovering the redo data received from the
primary database.

Logical standby
Contains the same logical information as the primary database, although the physical
organization and structure of the data can be different. It is kept synchronized with the
primary database by transforming the data in the redo logs received from the primary
database into SQL statements and then executing the SQL statements on the standby
database.
Snapshot standby
A fully updatable standby database. Like a physical or logical standby database, a
snapshot standby database receives and archives redo data from a primary database.
Unlike a physical or logical standby database, a snapshot standby database does not
apply the redo data that it receives.

This document details the steps for setting up the first of these types, a physical standby
database.

Note: Logical standby databases are not supported with Oracle E-Business Suite standard
functionality.

Snapshot standby databases should be used with caution as the data will be out of sync with the
primary as the redo data received on a snapshot standby is not applied. Therefore, using a
snapshot standby as a disaster recovery presents a certain risk, and should be used with caution.

1.2 Oracle Data Guard

Oracle Data Guard is a set of services that create, manage, and monitor one or more standby
databases to enable a primary database to survive disasters and data corruption. If the primary
database becomes unavailable due to a planned or an unplanned outage, Oracle Data Guard can
switch a standby database to the primary role, minimizing the downtime.

Data Guard offers three modes of data protection:

Maximum Protection
This mode offers the highest level of data protection. Data is synchronously transmitted
to the standby database from the primary database, and transactions are not committed
on the primary database unless the redo data is available on at least one standby database
configured in this mode. If the last standby database configured in this mode becomes
unavailable, processing stops on the primary database. This mode guarantees no data
loss.

Maximum Availability
This mode is similar to the maximum protection mode, including no data loss. However,
if a standby database becomes unavailable (for example, due to network connectivity
problems), processing continues on the primary database. When the fault is corrected,
the standby database is resynchronized with the primary database. If there is a need to
failover before the standby database is resynchronized, some data may be lost.
Maximum Performance
This mode offers slightly less data protection on the primary database, but higher
performance than maximum availability mode. In this mode, as the primary database
processes transactions, redo data is asynchronously shipped to the standby database. The
commit operation on the primary database does not wait for the standby database to
acknowledge receipt of redo data before completing write operations on the primary
database. If any standby destination becomes unavailable, processing continues on the
primary database, and there is little effect on primary database performance.

1.3 Oracle Data Guard Broker

The Oracle Data Guard broker is a distributed management framework that automates and
centralizes the creation, maintenance, and monitoring of Data Guard configurations. It logically
groups the primary and standby databases into a broker configuration that allows the broker to
manage and monitor them together as an integrated unit.

The Data Guard broker consists of the following three components:

Data Guard graphical user interface (GUI) through Enterprise Manager

Data Guard command-line interface (DGMGRL)
 Data Guard monitor

The broker automates and simplifies the following operations:

Standby database creation

The Enterprise Manager wizards automate and simplify the steps required to create a
configuration with an Oracle database on each site, including creating the standby
control file, online redo log files, data files, and server parameter files.

Note: This document contains specific steps to set up a physical standby with Oracle E-
Business Suite for both Oracle RAC and non-RAC.

Role transitions
This simplifies the switchover and failover process, including automatically setting up
redo transport and log apply services, and automating failover.

Note: Fast-start failover is currently not supported with Oracle E-Business Suite. For
further details, refer to Appendix E: Using Data Guard Broker (DGMGRL) to Manage
Standby Databases.

Monitoring
This provides continuous monitoring of the configuration health, database health, and
other runtime parameters.

Section 2: Before You Start

This section provides information on design considerations and software prerequisites. It is

based on the classic/traditional architecture of an Oracle E-Business Suite 12.2 Data Guard
configuration, where the systems are replicated across the primary and standby sites, as seen in
the diagram below:

This traditional approach is described in more detail in 5.1 Configuration 1: Standby

Application Tier Connected to Standby Database.

An alternative approach is to have all the application tiers connected to the primary (for
patching purposes). This configuration is further described in 5.2 Configuration 2: Standby
Application Tiers Connected to Primary Database. The advantages of this approach is that in the
event of a failover or switchover, you just need to connect the standby application tiers to the
standby database.

With configuration-1, when adop cycle is completed in primary, it is need to be applied in the
standby in hotpatch mode with nodatabaseportion option as described in section-7. The
advantage of configuration-2 over it is that an adop patch has to be applied only in primary.
There is no need to apply the patch in the standby site. In either case there is no need to recreate
the standby application tier.If a failover event occurs during an adop patching cycle, then with
both the configuration, adop patching cycle has to be aborted and then need to be reapplied.

As a conclusion, Configurtion-2 is a best approach but it requires disk replication from primary
admin node to standby admin node.

2.1 Design Considerations

This note assumes you already have a secondary site for your standby environment. In general,
the secondary data center should:

Be physically separate from the primary environment to protect against local and
regional disasters.
Have sufficient network bandwidth between the data centers to handle the peak redo and
concurrent manager output: report, log and output files.
Have reliable and efficient network services to the primary data center, to the standby
data center, and to the user locations.
Ideally mirror the number and type of the production machines, though fewer DR
machines could be used with reduced performance.

You should be familiar with the Oracle 12c (12.1.0.2) database server and have at least a basic
knowledge of standby database configurations.

Refer to the following documentation for further information:

Oracle Database Administrator's Guide 12c Release 1 (12.1), Part No. E17636-21
Oracle Data Guard Concepts and Administration 12c Release 1 (12.1), Part No.
E17640-15

In addition, you should be familiar with the following Oracle E-Business Suite Release 12
documentation:

My Oracle Support Knowledge Document 1383621.1, Cloning Oracle E-Business Suite

Release 12.2 with Rapid Clone
My Oracle Support Knowledge Document 1679270.1, Cloning Oracle E-Business Suite
Release 12.2 RAC Enabled Systems with Rapid Clone
2.2 Software Prerequisites

This document assumes the following minimum software versions:

Software Applicable
Additional Information
Component Version(s)
This document was developed using a freshly installed
database from an Oracle E-Business Suite Release 12.2 Rapid
Oracle E-
12.2.4, 12.2.5 Install with the prerequisite patches listed in the "Prerequisite
Business Suite
Tasks" section of chapter 17 of Oracle E-Business Suite 12.2
Setup Guide.
Oracle Database
12.1.0.2
12cRelease 1

Note: The standby and primary systems must use the same database and Oracle E-Business
Suite versions.

This document refers to the following top-level directories:

Directory Purpose
RDBMS_ORACLE_HOME Oracle 12c Release 1 (12.1.0.2) Database ORACLE_HOME
APPL_TOP Directory that contains the application product directories and files
COMMON_TOP
Directory that contains directories and files used across application
products
OracleAS 10.1.2
ORACLE_HOME ORACLE_HOME installed by Oracle E-Business Suite on application tier
FMW_HOME FMW_HOME installed by Oracle E-Business Suite on application tier
INST_TOP Directory that contains application instance directories and files

Note:
For Oracle RAC configurations: Refer to My Oracle Support Knowledge Document
1679270.1, Cloning Oracle E-Business Suite Release 12.2 RAC Enabled Systems with Rapid
Clone and ensure that you have applied all the required patches for RAC to RAC cloning.
For non-RAC configurations: Refer to My Oracle Support Knowledge Document
1383621.1, Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone and ensure that you
have applied all the required application and database cloning patches.

Prerequisite Patches:

For both Oracle RAC and non-RAC, download and apply the following Oracle E-Business Suite
patches:

For AD Delta 7:

R12.AD.C.DELTA.7 Patch 20745242 and R12.TXK.C.DELTA.7 Patch 20784380

One-off patches Patch 21841288 and Patch 21846184

Oracle E-Business Suite Release 12.2.4 Maintenance Pack Patch 17919161

For AD Delta 6:

R12.AD.C.DELTA.6 Patch 19197270 and R12.TXK.C.DELTA.6 Patch 19330775

One-off Patch 21878338

Oracle E-Business Suite Release 12.2.4 Maintenance Pack Patch 17919161

Section 3: Preparing the Primary Database for Standby Database Creation

At this point, you have a server available to create the physical standby. The top-level mount
points on each secondary site server are exactly the same as on their matching primary site
server. Ownership and permissions must be set appropriately for each mount point. You need to
have a mechanism in place for making remote file copies (or using disk replication), checking
network connectivity and appropriate system permissions. The steps for preparing the primary
database for standby database creation are divided into the following steps:

3.1 Enable Forced Logging

3.2 Configure Oracle Net Communication To and From Standby System
3.3 Set Up Secure Connections
3.4 Set Primary Database Initialization Parameters
3.5 Enable Archive Logging on Primary System
3.6 Add Standby Redo Logs
3.7 Invite Communications From Standby
3.8 Gather Temporary File Information
3.9 Run the Application Tier and Database Pre-clones
3.10 Copy APPL_TOP and Oracle E-Business Suite Technology Stacks to the Application Tier
in Standby Environment

3.1 Enable Forced Logging

To protect against unlogged direct writes in the primary database that cannot be propagated to
the standby database, turn on FORCE LOGGING at the primary database before performing data file
backups that will be used to create the standby database.

Use the following SQL command to set FORCE LOGGING on the primary database:
SQL>ALTER DATABASE FORCE LOGGING;

Note: This statement may take a considerable amount of time to complete as it must wait for all
unlogged direct write I/O operations to finish.

3.2 Configure Oracle Net Communication To/From Standby System

The primary and standby databases must be able to communicate using Oracle Net. This means
that on the standby, a listener needs to be running in order to handle incoming requests from the
primary. In addition, TNS aliases must be created on both the primary and standby
systems. Configure ifiles under the TNS_ADMIN directory for the aliases and
listener. Although you can use either a service model (dynamic registration) or SID (static
registration) model, note that Autoconfig only uses static registration and so this is the approach
used throughout this document.

Standby Listener

This listener only runs while the server is hosting a standby database. On switchover/failover,
etc., the standard AutoConfig listener is used. Use the same structure as the AutoConfig listener,
substituting different values for port, host and listener name. See Appendix A: Oracle Net
Files for an example.

TNS Aliases

The aliases are used by the fal_client/server init.ora parameters to enable two-way
communication between the primary and standby. The fal_client alias is the connect string to
the standby; the fal_server alias is the reverse - the connect string to the primary. Refer
to A1 of Appendix A: Oracle Net Files for an example.

Note For Oracle RAC Configurations: The TNS alias requirements are different. Refer
to A2 of Appendix A: Oracle Net Files for an example.

3.3 Set Up Secure Connections

Oracle Data Guard uses Oracle Net sessions to transport redo data and control messages
between the members of a Data Guard configuration. These redo transport sessions are
authenticated using either the Secure Sockets Layer (SSL) protocol or a remote login password
file. Use the following command to create a password file:
$ cd <RDBMS_ORACLE_HOME>/dbs
$ orapwd file=orapw <SID> password= <SYS password> entries= <max privileged users> ignorecase=y

Note For Oracle RAC Configurations: Create password files on all instances (this is the
approach used throughout this document). For further information, refer to My Oracle Support
Knowledge Document 1367293.1, Enabling SSL or TLS in Oracle E-Business Suite Release
12.2 for the database SSL setup.

To complete the implementation of the password file, add the

parameter remote_login_passwordfile to your init.ora file as described in the next step.

3.4 Set Primary Database Initialization Parameters

This section defines the initialization parameters that control redo transport services while the
primary database is in the primary role. Some minimal changes will be required when a role
transition occurs and the primary and standby databases change roles.

Note: This document uses a static init.ora include file to record parameters. If you are using an
spfile, disregard the ifile actions and use the appropriate "alter system" command to make the
necessary changes.

Define an archive log destination directory on the file system. Then add parameters to your
database init.ora file by using the ifile found at <RDBMS_ORACLE_HOME>/dbs/<CONTEXT_NAME>_ifile.ora.

Primary database: primary role initialization parameters

The table below lists the parameters relating to archive destinations and transmission of redo
data to the standby system.

Parameter
log_archive_dest_1
log_archive_dest_2
Description
Archives redo data generated on primary database to the local file
system.
Transmits redo data to the remote physical standby destination.

Options used:

The Standby Service Name. This is a service_name

SERVICE : on the primary site that points to the Standby
database.
Redo Log information can be transmitted in one of
two ways from primary to physical standby either
LGWR :
LGWR or ARCH process. LGWR is used in this
document.
Specifies that network I/O is to be performed
asynchronously for the destination. You can
optionally specify a block count (from 0 to 102,400)
ASYNC :
that determines the size of the SGA network buffer
to be used. Please refer to the example code block
below.
The minimum number of seconds before the log
REOPEN : writer process (LGWR) should try again to access a
previously failed destination.

log_archive_dest_state_2
log_archive_format
log_archive_min_succeed_dest
log_archive_config
db_unique_name
fal_server
fal_client
standby_file_management
db_file_name_convert,
log_file_name_convert
Remote_login_passwordfile
The maximum number of reopen attempts before
MAX_FAILURE: the primary database permanently gives up on the
standby database.
Specifies the number of seconds the log writer
process on the primary system waits for status from
NET_TIMEOUT:
the network server (LNS n) process before
terminating the network connection.
When set to ENABLE allows redo transport services to transmit
redo data to the specified destination. Set this value to ENABLE
on primary site to send archive log files automatically to standby.
Used to specify the filename format when archiving redo log files.
Will use system defaults if not set.
Defines the minimum number of destinations that must succeed in
order for the online log file to be available for reuse.
Enables or disables the sending of redo logs to remote destinations
and the receipt of remote redo logs, and specifies the unique
database names (DB_UNIQUE_NAME) for each database in the Data
Guard configuration.
Unique Name to identify the primary and standby (For example,
'dg12' for primary and 'dg12s' for physical standby).
Specifies the TNS network service name that the standby database
should use to connect to the FAL server process. FAL Server is
Fetch Archive Log (FAL) Server which services requests for
archive redo logs from FAL clients running on multiple standby
databases. Set this parameter to primary database service
name dg12 (for example) to request missing archived redo log
files if primary is unable to send the missing log files
automatically.
Specifies the TNS network service name that the primary database
should use to connect to the standby.
Set to AUTO. Whenever data files are added or dropped from
primary database, corresponding changes will be made
automatically to the standby.
Specify the path name and file name location of data files and redo
log files. The db_file_name_convert parameter does not need to be set
when the directory structures are same on the primary and standby.
Although, log_file_name_convert should be set to dummy values if
you are using same directory structure to enable redo log clearing.
This parameter checks specifies whether Oracle checks for a
password file. Since we are using password authentication, this
parameter should set to exclusive as in the example code block
 below..

For further explanation of the initialization parameters, refer to Oracle Data Guard Concepts
and Administration 12c Release 1 (12.1).

The configuration examples used throughout this document use the names shown in the
following table.

Primary Physical Oracle RAC Oracle RAC

Database Standby Primary Standby
Oracle Net Service
Name dg12 dg12s prod stdby
DB_NAME dg12 dg12 prod1 and prod2 prod1 and prod2
SID dg12 dg12 prod1 and prod2 prod1 and prod2
DB_UNIQUE_NAME dg12 dg12s prod stdby

Note: The database SID must be the same on both the primary and physical standby databases.

The following example shows the relevant initialization parameters of the primary database:
db_unique_name = dg12 ---- You need to change this to the standby db_unique_name (such as dg12s) when you copy this
file to the physical standby.
log_archive_dest_1 ="LOCATION=/arch1/dg12/MANDATORY"
log_archive_dest_2 ="SERVICE=dg12s LGWR ASYNC=20480 DB_UNIQUE_NAME=dg12s OPTIONAL REOPEN=15 MAX_FAILURE=10
NET_TIMEOUT=30"
log_archive_config='dg_config=(dg12,dg12s)'
# log_archive_dest_state_2 = defer
# log_archive_dest_state_2 = enable
log_archive_min_succeed_dest = 1

standby_file_management = AUTO

Remote_login_passwordfile = exclusive
log_archive_format = <name>%s_%t_%r. <ext> ---- Or you can just leave it set to the default.

# db_file_name_convert: ---- you do not need to specify this if you use the same directory structure
# log_file_name_convert: ---- you do not need to specify this if you use the same directory structure
fal_server = dg12
fal_client = dg12s
log_file_name_convert = xx,xx ---- Specify dummy values to trigger log clearing

Note For Oracle RAC Configurations:When using a shared file system, in order to prevent
overwriting archive log files for different nodes, you need to use a specific format when naming
the archive logs on each node.This is configured in the <instance>_<node>_ifile.ora file on each of
the nodes that form the primary Oracle RAC
system. The <instance>_<node>_ifile.ora configuration assumes that prod and stdby are the service
names/db_unique_names on the primary and standby respectively.
db_unique_name=prod
log_archive_dest_1='LOCATION=<ORACLE_HOME>/dbs/arch/prod MANDATORY'
log_archive_dest_2='service=stdby valid_for=(online_logfiles,primary_role) db_unique_name=stdby LGWR ASYNC=20480
OPTIONAL REOPEN=15 NET_TIMEOUT=30'
log_archive_dest_state_1 = enable
log_archive_dest_state_2 = enable
fal_server=prod
log_archive_format=prod1_%s_%t_%r.log
db_file_name_convert='<SHARED DATA LOCATION ON PRIMARY>','<SHARED DATA LOCATION STANDBY>'
log_file_name_convert='<Shared Log Files Location on Primary>','<Shared Log Files Location on StANDBY>'
standby_file_management=auto
3.5 Enable Archive Logging on Primary System

Ensure that archiving is enabled on the primary system.

3.6 Add Standby Redo Logs

Redo data transmitted from the primary database is received by the remote file server (RFS)
process on the standby system. The RFS process writes the redo data to archived log files or
standby redo log files. Redo data can be applied either after the redo is written to the archived
redo log file or standby redo log file, or, if real-time apply is enabled, directly from the standby
redo log file as it is being filled. Standby redo logs are required if you want to use, for example,
maximum protection and maximum availability levels of data protection, or real-time apply.

A best practice is to add them to both the primary and the standby database so that switchover
between the environments is quicker and easier. In this case, you will add them to production, so
that they are in place and will be cloned to the standby.There are two considerations:

On the primary: The standard Oracle recommendation is to multiplex redo logs on the
primary when using NORMAL redundancy, but not when using ASM with high
redundancy.

On the standby: Standby redo log files can be multiplexed using multiple members,
thereby improving reliability over archived log files. However, multiplexing redo logs
adds more IO overhead that could impact the standby's redo apply rate as multiplexing
generates N*3 writes write IOs (where N is the number of multiplex redo logs you
have). This is the reason why some references state "Do not multiplex the standby redo
logs". Clearly, there are several factors to consider and therefore it is highly
recommended that you test whichever solution you decide to implement.

As a general rule, follow the same best practice as for the primary for online redo logs: create
standby redo log files with the same sizes, groups and number of members as on the primary.

Standby redo logs still count towards the total allocation of redo log groups and members,
which are limited by the database MAXLOGFILES and MAXLOGMEMBERS parameters
respectively. It is therefore important that you size the files accordingly.

To create standby redo log groups, as the ORACLE user on the primary database server, connect
to SQL*Plus as sysdba and use the following command as a template for each standby redo log
group that you will create:
SQL>alter database add standby logfile group N (<log_file>) size NNN;

For an example, refer to Appendix C: Example Standby Database Commands.

Note: For Oracle RAC Configurations: Ensure that the standby logs are created for each redo
thread. For an example, refer to Appendix C: Example Standby Database Commands.

3.7 Invite Communications from the Standby

Oracle E-Business Suite Release 12 has a new security feature that restricts remote connections
to the database for clients that are not registered on the system, which is enabled by default.

When this option is enabled, any additional machines that require direct access to the Oracle E-
Business Suite database (using SQL*Plus, SQL*Navigator, etc.) will need to be 'Registered
Nodes' to explicitly obtain access. This is achieved by setting the invited nodes value
in sqlnet.ora file. If you have enabled invited nodes in sqlnet.ora, then the standby host needs to
be added to the list of nodes. You can perform this step using OAM. Refer to Appendix B:
Using Oracle Applications Manager to Register Standby Database for detailed steps.

3.8 Gather Temporary File Information

You will need to manually recreate your temporary files on the standby database. Gather the
required data now from the primary database with the following SQL*Plus query:
SQL>select file_name, bytes from dba_temp_files;

Save the output for use in a later step. For an example of the output, refer to Appendix C:
Example Standby Database Commands.

3.9 Run the Application Tier and Database Tier Pre-clone Scripts

Note:
Ensure that the applications ADOP patching cycle is complete.
Ensure the PERL5LIB environment variable includes the
path <RDBMS_ORACLE_HOME>/perl/lib/5.14.1/CGI in the CONTEXT_FILE and run AutoConfig on the
database tier. After the AutoConfig completes successfully, runadpreclone.pl on the
database tier.

1. As the ORACLE user, run the database pre-clone utility on the primary database server.
$ cd $RDBMS_ORACLE_HOME/appsutil/scripts/<context_name>/
$ perl adpreclone.pl dbTier

Supply the APPSpwd when requested.

2. Log on to the run edition file system on the Weblogic Admin server node on the source
system as the APPLMGR user. Run the following preclone command on both the run
and patch edition file systems:
$ cd $INST_TOP/admin/scripts/
$ perl adpreclone.pl appsTier
 Note For Oracle RAC Configurations: Run these pre-clone scripts on all database and
application tiers hosting the WLS admin server.
Note For non- RAC Configurations: Run these pre-clone scripts on database and
application tiers hosting the WLS admin server.

3.10 Copy APPL_TOP and Oracle E-Business Suite Technology Stacks to the Application Tier on the Standby
Environment

Refer to the the following for the list of the directories that you need to copy:

For Oracle RAC systems, refer to My Oracle Support Knowledge Document

1679270.1, Cloning Oracle E-Business Suite Release 12.2 RAC Enabled Systems with
Rapid Clone.
For Oracle non-RAC systems, refer to My Oracle Support Knowledge < Document
1383621.1>, Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone.

If your operating system returns errors when copying open files, you may need to shut down
application tier services to successfully copy the APPL_TOP and Oracle E-Business Suite
technology stack software as follows:
$ cd $INST_TOP/admin/scripts/
$ adstpall.sh apps/<apps password>

Section 4: Creating a Physical Standby Database

4.1 Copy ORACLE_HOME and Database to the Standby Database Server

4.2 Generate a Standby Control File and Copy it to the Standby Database Server (optional)
4.3 Perform File-based Configurations on the Standby Database Server
4.4 Stop Database Listener on the Standby Database Server
4.5 Configure Oracle Net for Redo Transmission and Start the Listener
4.6 Modify Database init.ora Parameters on the Standby Server
4.7 Clone the Database, mount the Physical Standby and Start Processing Redo on the Standby
4.8 Start Shipping Redo from the Primary to the Standby Database Server
4.9 Verify Redo is Shipping
4.10 Add Your Temp Files to the Standby Database
4.11 Configure Oracle Data Guard Broker (optional)

4.1 Copy ORACLE_HOME and Database to the Standby Database Server

Copy the Oracle home file system to the standby database server. If you natively compile your
PL/SQL, be sure to include the file system directories holding the compiled objects. The
standard location for this is<RDBMS_ORACLE_HOME>/plsql/nativelib.

There are three choices for backing up or copying the database to the standby site:

Manual cold backup

With the database shut down, copy all the database files and redo log files to the standby
 site.

Manual hot backup

With the primary database open, put the tablespace in backup mode, and copy the data
files to the standby site.
RMAN
Use the duplicate database command. For further information refer to Appendix D: Using
RMAN to Create Physical Standby Database.

If you use RMAN to perform your backup, you do not need to place the tablespaces in 'hot
backup' mode or manually create your standby control file. Refer to the "Overview of RMAN
Database Duplication" section in the Oracle Database Backup and Recovery User's Guide for
more details. RMAN restores a backup control file, and copies all necessary database files and
archived redo logs over the network to the standby host. However, while RMAN recovers the
standby database, it does not place it in manual or managed recovery mode.

Note For Oracle RAC Configurations only:

To create the RAC standby this document uses RMAN duplicate database for standby from
active database.The benefit of using this command is that it only mounts the database in
standby mode.
Standby creation steps using RMAN are described in appendix D for both non-RAC as
well as RAC systems.

4.2 Generate a Standby Control File and Copy it to the Standby Database Server (optional)

Note: This is an optional step. Skip this step if you used RMAN to copy the database.

If the backup procedure required you to shut down the primary database, create the control file
for the standby database.

You will need to recover past the time the control file is created, so you must switch logs and
note the new log number.
SQL>alter database create standby controlfile as <directory>/<controlfile name>;
SQL>alter system switch logfile;
SQL>select thread#, sequence#-1 from v$log where status = 'CURRENT';

Copy the control file to the standby database server.

Note the thread# and sequence# for later use. You will only be able to open the standby database
after this log has been applied on the standby.

4.3 Perform File-based Configurations on the Standby Database Server

After the database software copies are complete (which can be done before the database files
have finished copying), log into the standby database server as the ORACLE user and execute the
following commands to update the file system configuration for the new environment. As the
environment scripts have not been set up yet, you will need to manually resolve the reference
to <RDBMS_ORACLE_HOME>.
$ cd <RDBMS_ORACLE_HOME>/appsutil/clone/bin
$ perl adcfgclone.pl dbTechStack

Note For Oracle RAC Configurations only: Perform the following steps:
Run adclonectx to create the context file and then run the following command on all
nodes:
$ perl adclonectx.pl contextfile=<Primary Node Context File>
template=<RDBMS_ORACLE_HOME>/appsutil/template/adxdbctx.tmp
pairsfile=<RDBMS_ORACLE_HOME>/appsutil/clone/pairsfile.txt initialnode

Answer each of the questions. When prompted, use the initial node option
for adclonectx on all nodes, otherwise it will prompt for the "Live RAC node", which will
not be available at this time.
Modify the initialization parameter files so that they point the correct thread and have
the correct diagnostic destination ensure that utl_file_dir points to the standby context
directory. On completion, run the following command:
$ perl adcfgclone.pl dbTechStack <Context file created in the step above>

Your ORACLE user environment scripts are now ready to use. Source them so that they configure
the environment for the next steps, using the appropriate OS command.

For example, in sh or ksh on UNIX:

$ cd <RDBMS_ORACLE_HOME>
$ . <STNDBY_CONTEXT>.env

If you have implemented native PL/SQL compilation, set up an rsync job from the primary
database server to the standby database server for the file system directories holding the
compiled objects. The standard location for this is<RDBMS_ORACLE_HOME>/plsql/nativelib.

4.4 Stop Database Listener on the Standby Database Server

Adcfgclone in step 4.3 starts the database listener. This needs to be stopped as the standby
database listener is not yet fully configured. As the ORACLE user on the standby database server,
enter the following command:
$ lsnrctl stop <listener name>

4.5 Configure Oracle Net for Redo Transmission and Start the Listener

1. As the ORACLE user, copy the listener_ifile.ora and <CONTEXT_NAME>_ifile.ora from the
primary server's <TNS_ADMIN> directory to the standby server's <TNS_ADMIN> directory. As part
of the copy, rename the primary<CONTEXT_NAME>_ifile.ora to the
standby's <STNDBY_CONTEXT>_ifile.ora matching the spelling and case used for the file name
in the last line of the standby server's tnsnames.ora file.
 2. In the <STNDBY_CONTEXT>_ifile.ora, ensure the entry for the standby service's host parameter
holds the standby database host name and the FAL server host name is the primary host
name.
3. In the listener_ifile.ora file, ensure that the host entry for the standby service points to
the standby database host. Refer to Appendix A: Oracle Net Files for example TNS
entries.
4. As the ORACLE user, start the database listener for the standby:
$ lsnrctl start <standby service name>

4.6 Modify Database init.ora Parameters on the Standby Server

As the ORACLE user on the standby database server, create an ifile for the standby database from
the one you created earlier for the primary database:
$ cd <ORACLE_HOME>/dbs
$ cp <CONTEXT_NAME>_ifile.ora <STNDBY_CONTEXT>_ifile.ora

Update the following parameters:

DB_UNIQUE_NAMEshould be updated to a name different from the primary: for example, dg12s
LOG_ARCHIVE_DEST_2should point to the standby service. This is required when the standby
is switched to the primary and ships redo to new standby.
o For example: LOG_ARCHIVE_DEST_2 for 'service=dg12s ASYNC REGISTER
VALID_FOR=(online_logfile,primary_role) DB_UNIQUE_NAME=dg12'

Note: If you use RMAN duplicate you can include these database initialization parameter
changes in the command. It will create an spfile which also contains entries for the primary (so
it needs to be cleaned up, but you can make these modifications in that file).

If you are using an spfile and are therefore not using the AutoConfig generated init.ora, make
the following additional changes:

diagnostic_dest to <RDBMS_ORACLE_HOME>/admin/<STNDBY_CONTEXT>
utl_file_dir for context specific directories

Finally, add an entry for the standby control file you created on the primary and copied to this
server:
control_files = <control file directory>/<standby control file>, <control file directory>/<standby control file>

4.7 Clone the Database, mount the Physical Standby and Start Processing Redo on the Standby

Note For Oracle RAC Configurations only:

Create the physical standby RAC database using the RMAN duplicate command.
 For instructions, refer to Appendix D: Using RMAN to Create a RAC Physical Standby
Database.

Note: Ensure that the password file created in 3.3 Set Up Secure Connections exists
under $ORACLE_HOME/dbs.

As the ORACLE user on the standby database server, perform the following after the database copy
has completed:

1. Mount the standby database. Connect to SQL*Plus as sysdba and issue these commands:

Note: Skip this step if you used RMAN for standby creation.
SQL>startup nomount pfile= <RDBMS_ORACLE_HOME>/dbs/init <primary SID>.ora | spfile
SQL>alter database mount standby database;

2. Put the standby database in 'managed recover' mode.

SQL>alter database recover managed standby database disconnect from session;

4.8 Start Shipping Redo from Primary to Standby Database Server

As the ORACLE user on the primary database server, set log_archive_dest_state_2 to enable in the
database initialization file. This can be done using the following SQL command:
SQL> alter system set log_archive_dest_state_2='enable' scope=both sid='<Primary SID>';

4.9 Verify Redo is Shipping

Check to see that the database is shipping redo, by connecting to the primary database and
forcing a log switch:
SQL>alter system switch logfile;

While still on the primary, use the following command to review the status of the archive
destinations. Ensure that he most recently archived redo log file is the same at each
destination. If it is not, a status other than VALID may identify the error encountered during the
archival operation to that destination. If not you should resolve this before continuing.
SQL>select * from v$archive_dest_status where status != 'INACTIVE';

On each database server, the following query will show which logs have been sent/received and
applied:
SQL>select sequence#, applied, to_char(first_time, 'mm/dd/yy hh24:mi:ss') first from v$archived_log order by
first_time;
On the standby database server, monitor the database alert log to review the recovery progress.

4.10 Add Your Temp Files to the Standby Database

Note: Skip this step if you used RMAN to create the standby.

To save time on failover, add your temp files to the standby database at this point using the
temporary file names and sizes collated during 3.8 Gather Temporary File Information.

To do this, you need to open the database in read-only mode. You will not be able to open the
database read-only until recovery has progressed past the time the control file was created;
recall that the log sequence number was noted in 4.2 Generate a Standby Control File and Copy
it to the Standby Database Server.

Execute the following commands:

SQL>alter database recover managed standby database cancel;
SQL>alter database open read only;
SQL>alter tablespace temp add tempfile ' <file spec>' size <size> reuse;
[enter as many lines as you have temporary data files]
SQL>shutdown immediate;
SQL>startup mount;
SQL>alter database recover managed standby database disconnect from session;

4.11 Configure Oracle Data Guard Broker (optional)

If you wish to use Oracle Data Guard broker to manage your Oracle E-Business Suite standby
database, follow steps outlined in E1 and E2 of Appendix E: Using Data Guard Broker
(DGMGRL) to Management Standby Databases.

Section 5: Configuring the Application Tier After the Standby Database has been Enabled

In Oracle E-Business Suite 12.1.3, an application patch on the primary would be copied to the
standby tier and the cloning scripts would be used to update the configuration.

In Oracle E-Business Suite 12.2, this is complicated by the run and patch file systems which
need to be kept in sync. The process described in this document takes the approach of patching
both the primary and standby application tiers. The alternative approach is to connect both the
primary and standy application tiers to the primary instance, which means that in the event of a
failover or switchover, you just need to disconnect the standby application tier from the primary
and connect it to the standby database.

This section explains the two possible methods to configure the standby application tiers:

5.1 Configuration 1: Standby Application Tier(s) Connected to the Standby Database

5.2 Configuration 2: Standby Application Tier(s) Connected to the Primary Database

5.1 Configuration 1: Standby Application Tier(s) Connected to the Standby Database

The following diagram shows the traditional configuration of a symmetrical disaster recovery
site, where the application tiers connect to the their respective instance (primary or standby).

Once the application tier software copies have completed, you need to run the cloning scripts to
clone the applications tiers, which will require a connection to the database. The standby
database will not be open and therefore you need to open the standby database in snapshot mode
and then execute adcfgclone to clone the application tier. Once this has completed, change the
standby back to standby mode.

5.1.1 Convert the Standby to a Snapshot Standby

1. The first step is to configure the physical standby database to use flashback logging for
flashback database operations.

2. A flashback database requires that the flashback logs reside in the flash recovery area
(FRA). Although the FRA may already be configured for your environment, the
flashback logs can significantly increase FRA usage. It is not unheard of to double the
size of the FRA to utilize flashback database and so you need to take this into account.
SQL>alter system set db_recovery_file_dest_size=32g scope=both;
SQL>alter system set db_recovery_file_dest='/u03/flash_recovery_area' scope=both;
SQL>alter system set db_flashback_retention_target=1440 scope=both;

Note: If you are using a pfile, add the following entries to the ifile located
in <contextname_ifile>.ora under $ORACLE_HOME/dbs:

o db_recovery_file_dest_size=32G
o db_recovery_file_dest='/u03/flas_recovery_area

o db_flashback_retention_target=1440

Substitute an appropriate destination size for your system.

3. Stop redo apply on the physical standby database:

SQL>alter database recover managed standby database cancel;

4. Turn on flashback logging:

SQL>alter database flashback on;

5. Convert the standby database into a snapshot standby database:

 Note For Oracle RAC Configurations only: Shutdown all other instances except one,
which is the one you will be using to execute the following command.
SQL>alter database convert to snapshot standby;

6. Shut down the snapshot standby database and startup the database (it will be opened for
read/write access):
SQL>shutdown immediate;
SQL>startup

Note For Oracle RAC Configurations only: Startup all the instances.

7. Remove the applications configuration.

$ sqlplus apps/<password>
SQL>exec fnd_conc_clone.setup_clean;
SQL>exec ad_zd_fixer.clear_valid_nodes_info;

8. Run AutoConfig on the database tier.

$ $ORACLE_HOME/appsutil/scripts/<CONTEXT_DIRECTORY>/adautocfg.sh

Note For Oracle RAC Configurations only: Run AutoConfig on all of the RAC
database nodes once, and then run it again on each node so that it updates
the sqlnet.ora with all the node names.

9. Add the application tier nodes to the $TNS_ADMIN/sqlnet.ora file on each of the standby
database nodes.

10. Reload the database listeners so that they reread the sqlnet.ora using the following
command:
$ lsnrctl reload <EBS Listener Name>

5.1.2 Configure the Standby Application Tier

1. Log on to the run edition file system on the standby system as the APPLMGR user and enter
the following commands:
$ cd <COMMON_TOP>/clone/bin
$ perl adcfgclone.pl appsTier

Note: In AD-TXK Delta 7, we recommend you clone the Application tier run and patch
file systems in a single operation, using the 'dualfs' option. Separate cloning of the run
and patch file systems will be deprecated in the future. If 'dualfs' option is used, step 2, 3
 and 4 given below are not required. For detailed instructions on configuring shared (and
non-shared) APPL_TOP systems, refer to Section 5.2 of My Oracle Support
KnowledgeDocument 1383621.1,Cloning Oracle E-Business Suite Release 12.2 with
Rapid Clone.

2. Run adpreclone.pl on the run edition file system in the standby system.

3. Copy the Oracle E-Business Suite application directories (<APPL_TOP>, <common_top>,Oracle

AS Tools 10.1.2 Oracle_Home) from the run edition file system to the patch edition file
system on the standby system.

Note: When copying files, ensure that the symbolic links are also created under <OracleAS
Tools 10.1.2 ORACLE_HOME> and <COMMON_TOP>.

4. Once you have cloned the run file system, you also need to clone the patch file system
from the run file system. Run the following command as the APPLMGR user on the patch
edition file system in the target standby system:
$ cd <COMMON_TOP>/clone/bin
$ perl adcfgclone.pl appsTier

Note: For detailed instructions on creating a standby application tier (either as a single
node or a master node in a multi node configuration), refer to My Oracle Support
Knowledge Document 1383621.1, Cloning Oracle E-Business Suite Release 12.2 with
Rapid Clone.

5. To configure multiple non-shared APPL_TOP application tiers:

1. Execute $INST_TOP/scripts/adpreclone.pl appsTier on both run and patch file systems
of the first/master node.
2. Copy the EBSapps directory for both the run and patch systems to the other nodes.
For example, the master node has fs1 as run and fs2 as patch, then
copy fs1/EBSapps and fs2/EBSapps to the other nodes.
3. Execute $COMMON_TOP/clone/bin adcfgclone.pl appsTier on both the run and patch file
system by selecting the add node option on the other node.

Note: For detailed instructions on configuring shared (and non-shared)

APPL_TOP systems, refer to Section 5.2 of My Oracle Support
Knowledge Document 1383621.1,Cloning Oracle E-Business Suite Release 12.2
 with Rapid Clone.

5.1.3 Revert the Physical Standby Database to its Original State

1. Change the standby database from snapshot mode back into standby mode using the
following commands:

Note For Oracle RAC Configurations only: Shutdown all of the other instances before
running the following commands:
SQL>startup mount force;
SQL>alter database convert to physical standby;
SQL>shutdown immediate;
SQL>startup nomount;
SQL>alter database mount standby database;

Note For Oracle RAC Configurations only: Restart all of the instances.

2. Enable redo log apply:

SQL>alter database recover managed standby database using current logfile disconnect;

3. On the primary database, issue the following statement to re-enable archiving to the
physical standby database:
SQL>alter system set log_archive_dest_state_2=enable scope=both;

5.2 Configuration 2: Standby Application Tier(s) Connected to the Primary Database

The following diagram shows all the standby application tiers are connected to the primary
database, which simplifies patching as all tiers will participate in a patching cycle. Note that the
services on the standby applications tiers will be disabled. Also note that the standby master
node is separate to this group and will need to be be replicated separately from the primary
master node. On switchover or failover, the standby application tiers will need to be connected
to the standby database, which will become the new primary server following a role
transition. This simplifies patching as compared to the previous approach, but the trade-off is
that there are several configuration steps that need to be performed.

This section describes the primary and standby configuration of an example two node RAC with
3 non-shared multiple application tiers.

Primary Site
 prdbnode1 stdbnode1
Database Tier (RAC)
prdbnode2 stdbnode2

stdby-node1 (Admin Node is in a passive stat

prim-node1 (Admin Node)
following a switchover/failover).
Application Tier
prim-node2 stdby-node2
prim-node3 stdby-node3

Note: In this configuration the Admin server node APPL_TOP should not be shared with any of
the slave nodes. The slave nodes can have shared APPL_TOP so long as they are not shared
with the Admin node either the primary or standby sites. The reason is that the admin server is
replicated to standby site and it makes the switch over faster and reduces the number of steps.

This procedure adds the standby application tiers to the primary database:

1. Configure the standby admin server application node(s) to use shared storage such that
the file systems can be replicated from primary site admin server node using disk
replication. Refer to the "Storage Replication" section of the Oracle Fusion Middleware
Disaster Recovery Guide for further information.

2. Configure the secondary nodes as follows:

Note: It is important that you familiarize yourself with the concepts and overview
discussed in My Oracle Support Knowledge Document 1383621.1, Cloning a Multi-
Node System section of Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone.

1. Run the pre-clone scripts on the primary application admin server node on both
the RUN (fs1 for example) and PATCH (fs2 for example) file systems. In the
table above, the admin server is hosted on prim-node1:
$ perl $INST_TOP/scripts/adpreclone.pl appsTier

2. Copy the EBSapps directories to the new standby application tiers (on the slave
nodes) from the RUN (under fs1) and PATCH (under fs2) file systems on the
primary application tier.

3. Run the clone scripts to add the standby application tier to the primary using the
add node option. During cloning, the admin server on the primary server needs to
be up and running to access both the run and patch file system.
$ perl $COMMON_TOP/clone/bin/adcfgclone.pl appsTier
 Use the following questions and answers:
"Target System File Edition type [run]:" answer 'run'
"Do you want to add a node (yes/no):" answer 'yes'

4. At this point, all the standby application tiers will have been added to the primary
database. To copy the Weblogic domain configuration to newly added nodes,
start any of the mananged servers and then stop it.
$ admanagedsrvctl.sh start <any manager server for example oacore_server4>

5. Disable the services on standby application tier nodes in order to avoid having
the services startup whenever cutover is run . In the run and patch file system
context files, set the services to a disabled state on all of the standby slave nodes.
<oa_service_group_status oa_var="s_root_status">disabled</oa_service_group_status>
<oa_service_group_status oa_var="s_web_admin_status">disabled</oa_service_group_status>
<oa_service_group_status oa_var="s_web_entry_status">disabled</oa_service_group_status>
<oa_service_group_status oa_var="s_web_applications_status">disabled</oa_service_group_status>
<oa_service_group_status oa_var="s_batch_status">disabled</oa_service_group_status>
<oa_service_group_status oa_var="s_other_service_group_status">disabled</oa_service_group_status>

6. Run AutoConfig on all standby application tier slave nodes after disabling the
services.

At the end of these steps, the standby application tier will have been added to the primary
database, with the services in disabled state. When a role transition occurs you will need to
enable the services as described in the next section.

Section 6: Performing Role Transitions

A database can operate in either a primary or standby role; these roles are mutually exclusive.
Oracle Data Guard allows you to change these roles dynamically using SQL commands and
supports the following transitions:

Switchover
Allows the primary database to switch roles with one of the standby databases. There is
no data loss during a switchover. After a switchover, each database continues to
participate in the Oracle Data Guard configuration with its new role.

Failover
Changes a standby database to the primary role in response to a primary database failure.

Note For Oracle RAC Configurations: Only one primary instance should be mounted during a
switchover, all other nodes must be shut down and restarted on completion. On failover, only
one standby instance should be mounted (which becomes the new primary node) and then the
other nodes restarted on completion.
Several of the application configuration steps are common to each of these transitions and are
discussed in the final part of this section:

The following role transitions are discussed:

6.1 Perform a Switchover

6.2 Perform a Failover
6.3 Perform a Switchback to the Primary Following a Switchover/Failover
6.4 Configure the Application Tier After a Role Transition (Switchover/Failover/Switchback)

Note: If the servers are still accessible, stop any running application services before initiating a
switchover or switchback.

6.1 Perform a Switchover

Note: If you are using Data Guard broker to manage the standby database, follow the steps
provided in section E3 Role Transitions section of Appendix E: Using Data Guard Broker
(DGMGRL) to Manage Standby Databases, then proceed to step 6.1.6 Complete database
configurations.

A switchover is typically used to reduce the primary database downtime during planned outages,
such as operating system or hardware upgrades, or rolling upgrades of the Oracle database
software and patch sets. A switchover takes place in two phases. In the first phase, the existing
primary database undergoes a transition to a standby role. In the second phase, a standby
database undergoes a transition to primary. In this case, the primary site is accessible and
involved in the switchover.

Note: Complete the ADOP patch cycle before performing a switchover.

Sections 3, 4, and 5 define how to set up your environments in an attempt to minimize the
number of parameter changes. This section assumes you have configured your parameter files
accordingly

6.1.1 Prepare for a switchover to the standby server

1. Verify the primary database instance is open and the standby database instance is
mounted.

2. Shut down all the application tier services on all nodes on the primary site.
$ adstpall.sh <appsuser>/<appspass>

3. Verify there are no active users connected to the database. Shut down all the sessions on
the primary database.
 4. Use the following command on both the primary and standby databases to ensure that
the last redo data transmitted from the primary database has been applied on the standby
database. If necessary, perform a log file switch before proceeding further:.
SQL>select sequence#,applied from v$archived_log;

5. Check whether the primary is ready for a switch. Query the switchover_status column of
the v$database fixed view to determine whether the database is ready to switch modes:
SQL>select switchover_status from v$database;

If this query returns "TO STANDBY", then the environment is ready to switch. If it
returns "ACTIVE SESSIONS", then the switch command should be used with the
'session shutdown' option.

6.1.2. Initiate a switchover on the primary database

Note For Oracle RAC Configurations only: Only one primary instance should be mounted
during a switchover; all others must be shut down. After the switchover, set
LOG_ARCHIVE_DEST_STATE_2 on all of the new primary instances before starting them

Connect as sysdba and issue the following SQL command:

SQL>alter database commit to switchover to physical standby with session shutdown;

When this statement has completed, the primary database will have been converted to a standby
database. During the statement's execution, the current control file is backed up to the current
SQL session's trace file - this makes it possible to reconstruct the control file should the need
arise.

6.1.3 Shut down and mount the primary database

To complete the transition, the database must be shut down and restarted in a mounted state.
Change the value of LOG_ARCHIVE_DEST_STATE_2 to defer, in addition, recovery can be started in
preparation for transition.
SQL>shutdown immediate
SQL>startup nomount pfile=$ORACLE_HOME/dbs/init<SID>.ora | spfile
SQL>alter database mount standby database;
SQL>alter system set LOG_ARCHIVE_DEST_STATE_2=defer scope=both sid='*';
SQL>alter database recover managed standby database disconnect from session;

At this point in the switchover, both databases are standby databases.

6.1.4 Verify the switchover status on the standby server

As the ORACLE user on the standby-to-become-the-primary database server, verify that it is ready
to switch to the primary:
SQL>select switchover_status from v$database;

At this point, this should return a value 'TO PRIMARY'. Any other value, such as SESSIONS ACTIVE, NOT
ALLOWED should be investigated and corrected (as in step 6.1.2 above) before continuing.

6.1.5 Switch the selected standby database to a primary role

For the switchover process to occur, a standby database must be either mounted and in Redo
Apply mode, or open in read only mode with Redo Apply. Once it is mounted in an appropriate
mode, issue the following command to transition it to the primary role:
SQL>alter database commit to switchover to primary with session shutdown;

Adjust the network parameters on both database servers

As the ORACLE user on both the primary and standby database servers, change the host name in the
standby service definitions to point to the new standby server in both
the <CONTEXT_NAME>_ifile.ora and <STNDBY_CONTEXT>_ifile.ora (which are located in
the <TNS_ADMIN> directory)

On the current primary, change the LOG_ARCHIVE_DEST_STATE_2 to enable using the following
command:
SQL>alter system set LOG_ARCHIVE_DEST_STATE_2=enable scope=both sid='*';

To complete the transition, the database must be shut down and re-started:
SQL>shutdown immediate
SQL>startup pfile=$ORACLE_HOME/dbs/init<SID>.ora | spfile

6.1.6 Complete the database configuration

1. Connect to the new primary database using SQL*Plus as the APPS user, and execute the
following commands:
SQL>exec fnd_net_services.remove_system ('<SID>');
SQL>commit;
SQL>exec fnd_conc_clone.setup_clean;
SQL>exec ad_zd_fixer.clear_valid_nodes_info;

2. As the ORACLE user on the new primary database server, use AutoConfig to complete the
configuration for the primary operations, providing the APPS password when prompted:
$ cd <RDBMS_ORACLE_HOME>/appsutil/scripts/<context>
$ ./adautocfg.sh

3. When this completes, stop and start the listener on the new primary database server:
$ lsnrctl stop <SID>
$ lsnrctl start <SID>

4. On the new standby server, stop and start the listener for the standby services:
$ lsnrctl stop <standby service>
 $ lsnrctl start <standby service>

Note For Oracle RAC Configurations: Repeat steps 2-4 on each instance (each node
will have its own listener). Rerun AutoConfig on all nodes after completing steps 2-4 for
each instance to update all configuration files with all nodes in the cluster.

5. For applications-specific configurations, follow the steps in 6.4 Configuring the

Application Tiers After Role Transition (switchover, failover, switchback).

6.2 Perform a Failover

Note: If you are using Data Guard broker to manage standby databases follow the steps
provided in section E3 of Appendix E: Using Data Guard Broker (DGMGRL) to Manage
Standby Databases, then go to 6.2.7 Complete the database configuration.

You may need to failover to your standby site due to complete failure of the primary site. This
section describes the steps in the event of failure of the primary site. Sections 3, 4 and 5 define
how to set up your environments to minimize the parameters that need to be changed during fail
over. This remainder of the steps in this document assume that you have configured your
parameter files as described.

Performing a failover separates the standby database from the primary. You will need to create a
new standby database environment from the environment to which you failed over in order to
restore your disaster recovery protection.

6.2.1 Flush any unsent redo from the primary database to the target standby database

If the primary database can be mounted, it may be possible to flush any urgent archived and
correct redo from the primary database to the standby database. Ensure that redo apply is active
on standby server. Mount the database, but do not open it.
SQL>alter system flush redo to 'target_db_name';

This statement flushes any unsent redo from the primary database to the standby database, and
then waits for that redo to be applied to the standby database.

6.2.2 Verify that the standby database has the most recently archived redo log file for each primary database redo
thread

Query the V$ARCHIVED_LOG view on the target standby database to obtain the highest log sequence
number for each redo thread:
SQL>SELECT UNIQUE THREAD# AS THREAD, MAX(SEQUENCE#) OVER (PARTITION BY thread#) AS LAST from V$ARCHIVED_LOG;

6.2.3 Identify and resolve any archived redo log gaps

On the standby database server, connect as sysdba to the standby database. Query v$archive_gap to
determine whether there are missing archive logs:
SQL>select * from v$archive_gap;

If this query returns a row, this indicates that at least one archived redo log is missing from the
standby.

If you still have access to your primary database, you can determine the full name of the redo
logs by querying v$archived_log, using the low_sequence# and high_sequence# returned above, as
shown in the following commands:
SQL>select name from v$archived_log
where thread# = <thread# from above query>
and sequence# between <low_sequence# above> and <high_sequence# above>;

Locate the missing logs and copy them to the standby server's standby redo log destination.
Next register them using the following command:
SQL>alter database register physical logfile '<filespec/name on standby>';

Note: Only one gap at a time is reported in v$archive_gap. If you find a gap and resolve it, repeat
this process until no more gaps are reported.

6.2.4 Adjust standby archive destination status

On the standby database server, change the initialization parameter for the destination state of
the archive logs to be shipped from primary to standby from 'defer' to 'enable'. Make the
changes in<RDBMS_ORACLE_HOME>/dbs/<STNDBY_CONTEXT>_ifile.ora, commenting out the 'defer' line.
# log_archive_dest_state_2 = enable
log_archive_dest_state_2 = defer

6.2.5 Stop redo apply and finish applying all received redo data

When all available logs are present and registered on the standby, stop the redo apply:
SQL>alter database recover managed standby database cancel;

Finish the recovery session:

SQL>alter database recover managed standby database finish;

When that completes, convert the physical standby to a primary database role:
SQL>alter database commit to switchover to primary;
SQL>shutdown immediate;
SQL>startup pfile=?/dbs/init<SID>.ora

Note: You should back up this database without delay, as you cannot recover any changes made
after the failover without a fresh backup.
6.2.6 Update the TNS parameters on the new Standby Database.

As the ORACLE user on the new primary database server:

1. Update the HOST name in the standby service definition of

the <TNS_ADMIN>/<STNDBY_CONTEXT>_ifile.ora file to point to the new primary host.

6.2.7 Complete the database configuration

1. Connect to the new primary database using SQL*Plus as the APPS user, and execute the
following commands:
SQL>exec fnd_net_services.remove_system('<SID>');
SQL>commit;
SQL>exec fnd_conc_clone.setup_clean
SQL>exec ad_zd_fixer.clear_valid_nodes_info;

2. As the ORACLE user on the new primary database server, run AutoConfig and provide the
APPS password when prompted:
$ cd <ORACLE_HOME>/appsutil/scripts/<CONTEXT_NAME>
$ ./adautocfg.sh

3. When this completes, restart the listener on the new primary database server as follows:
SQL>lsnrctl stop <SID>
SQL>lsnrctl start <SID>

To complete the application-specific configuration, follow the steps in section 6.4 Configure the
Application Tier After Role Transition (Switchover, Failover, Switchback).

6.3 Perform a Switchback to the Primary Following a Switchover/Failover

6.3.1 Switch back to primary site after a switchover

After switchover to standby and maintenance is complete, you need to switch back to the
primary site. In this case, the pre-switchback configuration will be as follows:

Standby Site Primary Site

Primary database Standby database

Perform the following steps to switchback to the primary site:

1. Verify that the primary database at the standby site is open, and the standby
database at the primary site is mounted.
2. Verify all the redo logs are transferred to standby and are applied.
3. On the primary site, check whether the switchover_status from v$database is
showing: TO STANDBY.
 4. Shutdown all the services on the primary application tier.
5. On the primary database, issue the command:
SQL>alter database commit to switchover to physical standby;

6. Change the LOG_ARCHIVE_DEST_STATE_2 to defer at the standby site (primary database)

and set it to enable at the primary site (standby database).
7. Adjust the network configuration as mentioned in 6.1.5 Switch selected standby
database to primary role.
8. Shut down and mount the database as a standby at the standby site.
9. Start the recovery by issuing the following command at the primary site:
SQL>alter database commit to switchover to physical primary

10. Shut down and start up the database at the primary site. Complete the database
configuration by following 6.1.6 Complete database configurations.
11. Once the database has restarted, verify the redo log shipping. Refer to 6.1
Perform a Switchover for the commands.
12. Execute the database configuration steps mentioned in section 6.1.6 Complete
database configurations
13. Follow the steps in 6.4 Configure the Application Tier After Role Transition
(Switchover/Failover/Switchback).

6.3.2 Recreating the original primary database after failover

Once you have resolved the problem at the original primary site that necessitated the failover to
a standby database, you may be able to recover the primary system, apply the redo, and switch
roles. If not, you will need to recreate the primary database on the original primary site using the
following procedure:

1. Make a consistent backup of the active standby database at the standby site.
2. Restore the backup created at standby site on the primary database.
3. Run AutoConfig on both the database and application tiers.
4. Shut down and start up the database.
5. On the original primary site, create or modify the initialization parameter file
with appropriate values (as defined throughout this document) and, if necessary,
restart the database.
6. Create a new standby database at the original standby site by following the
instructions in Sections 3, 4 and 5.
7. For application-specific configurations, follow the steps in 6.4 Application Tier
 Configuration After a Role Transition (Switchover/Failover/Switchback).

6.4 Configure the Application Tier After Role Transition (Switchover/Failover/Switchback)

There are two methods explained in this document to configure application tiers - Configuration
1 and Configuration 2 as decribed in Section 5.

6.4.1 Configuring the Application Tiers after a role transition when the application tiers are
configured using Configuration 1.
6.4.2 Configuring the Application Tiers after a role transition when the application tiers are
configured using Configuration 2.

6.4.1 Configuring the Application Tiers after a role transition when the application tiers are configured using
Configuration 1.

6.4.1.1 Finish Oracle E-Business Suite configuration on the application tier

Note: Ensure that you have added all application node names to the $TNS_ADMIN/sqlnet.ora file on
all of the database nodes.

Once the primary database configuration has completed and its listeners have started, log in to
each new-primary application tier server as the APPLMGR user, and run the final configuration steps
on the run file systems providing the APPS password when prompted::
$ cd $INST_TOP/admin/scripts
$ ./adautocfg.sh

On the patch file system change to $AD_TOP/bin/ and run the following command:
$ perl adconfig.pl contextfile= <INST_TOP>/appl/admin/PATCH CONTEXT.xml -syncctx

Note: If you are not on AD.Detla.7 and TXK.Delta.7 or later , running AutoConfig will
complete but the the application tier services will fail to start due to database connectivity
issues. In this case, simply run AutoConfig a second time and then start the applications tier
services.

6.4.1.2 Update the Host Name in the FND_CONCURRENT_REQUESTS and FND_CONC_REQ_OUTPUTS tables

In order to synchronize your concurrent manager log and out directories, you must change the
host name in the fnd_concurrent_requests table to the standby server name:
SQL>update apps.fnd_concurrent_requests
set logfile_node_name = <new application tier node>,
outfile_node_name = <new application tier node>
where logfile_node_name = <old application tier node>
and outfile_node_name = <old application tier node>;

SQL>update apps.fnd_conc_req_outputs set file_node_name=<new applications tier node>

where file_node_name=<old applications tier node>;

If you do not synchronize your concurrent manager log and out directories, blank out the host
name in the fnd_concurrent_requests table to avoid network timeout errors:
SQL>update apps.fnd_concurrent_requests
set logfile_node_name = null,
outfile_node_name = null;

SQL>update apps.fnd_conc_req_outputs set file_node_name=' ';

Note: You must execute these changes before starting the concurrent managers, otherwise you
must add a where clause to restrict the update to only the old host names. If you let users on to
the system before this completes, the users will get errors when they try to access log or out files
that were generated on the old primary system.

6.4.1.3 Perform the cloning finishing tasks

Perform the following steps, which are documented in Section 4: Finishing Tasks of My Oracle
Support Knowledge Document 1383621.1, Cloning Oracle E-Business Suite Release 12.2 with
Rapid Clone.

1. Update the profile options. Rapid Clone updates only site level profile options. You must
manually update any other profile options set to instance specific values.
2. Update the printer settings. If the new cloned system needs to utilize different printers,
update the Target System with the new printer settings now.
3. Update Oracle Workflow configuration settings. Update the host and instance-specific
information used by Oracle Workflow.
4. Verify the APPLCSF variable is set correctly.

6.4.1.4 Direct users to the new system

The standby system should be available to your users as your new primary system. Direct your
users to the new URL.

6.4.1.5 Establish a new standby system

Perform this step only if you performed a failover. Failing over to the standby database (versus
switching over) separates it from the old primary. You must create a new standby environment
from this new system to again provide disaster recovery protection.

6.4.1.6 Re-point your CM log and out and native PL/SQL object directory rsync scripts (optional)

If you are keeping your concurrent manager log and out directories synchronized across the
environments, set up your rsync scripts to copy the files from the new primary server to the new
standby server.

6.4.2 Configuring the Application Tiers after a role transition when the application tiers are configured using
Configuration 2.

The following configuration steps need to be performed only on the slave application tier
nodes. In this configuration, the admin server needs to be replicated to one of the passive
application tier nodes on the standby.
6.4.2.1 Configure the Standby Admin Server Application Tier.

In this configuration, you need to replicate the primary admin server application tier. If
necessary (depending on the method of disk replication that you are using) for the standby WLS
Admin server, you may need to attach the storage to the new server.
1. On the Standby admin server, change the hostname and IP address
in /etc/hosts, /etc/sysconfig/network, and the /etc/sysconfig/network-scripts/ifcfg-
eth0 files as in the following examples:

In /etc/hosts add an entry with the primary admin node name and IP address in
the following format:
IPAddress prim-node1.domain prim-node1

In /etc/sysconfig/network:
NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=prim-node1.us.oracle.com

In /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0 ONBOOT=yes
BOOTPROTO=static
IPADDR=<New IP Address>
NETMASK=<netmask>
GATEWAY=<gateway>
TYPE=Ethernet

2. Restart the network services or reboot the server.

3. Modify the context variables in the context file and tnsnames.ora to connect to the
new primary database.

4. Configure password less SSH to each of the slave nodes from this nodes.
6.4.2.2 Modify the context variables in the context files on the run and patch filesystems on all of the slave application tiers to update the connectivity to
point to the new primary.

Note for Switchover: Run steps [1-6] on all slave nodes of both the primary and the standby so
that all the application tiers will be connected to the current primary database.
Note for Failover: Run steps [1-6] on all slave nodes only (as the primary application tiers are
not available).
1. In the context files, change s_dbhost and s_dbport to point to the first node of the current
primary RAC database.
from:
<dbhost oa_var="s_dbhost">prdbnode1</dbhost>
to:
 <dbhost oa_var="s_dbhost">stdbnode1</dbhost>

2. Mark the services as being enabled on the standby application tier(s):

<oa_service_group_status oa_var="s_root_status">enabled</oa_service_group_status>
<oa_service_group_status oa_var="s_web_admin_status">disabled</oa_service_group_status>
<oa_service_group_status oa_var="s_web_entry_status">enabled</oa_service_group_status>
<oa_service_group_status oa_var="s_web_applications_status">enabled</oa_service_group_status>
<oa_service_group_status oa_var="s_batch_status">enabled</oa_service_group_status> ******** Enable this if
you are using ICM on that node.
<oa_service_group_status oa_var="s_other_service_group_status">disabled</oa_service_group_status>

3. Change the JDBC context variables s_apps_jdbc_connect_descriptor and

s_apps_jdbc_patch_connect_descriptor to the new Database Nodes/SCAN Host in the context
file:
<jdbc_url
oa_var="s_apps_jdbc_connect_descriptor">jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)
(FAILOVER=YES)
(ADDRESS=(PROTOCOL=tcp) (HOST=<NEW SCAN NAME)(PORT=<NEW SCAN PORT>)))(CONNECT_DATA=(SERVICE_NAME=<Service
Name>)))</jdbc_url>

<patch_jdbc_url
oa_var="s_apps_jdbc_patch_connect_descriptor">jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)
(HOST=<New SCAN Host)(PORT=<New Scan port>))(CONNECT_DATA=(SERVICE_NAME=ebs_patch)
(INSTANCE_NAME=r122b1)))</patch_jdbc_url>

Note: If you have configured VIP instead of SCAN, change all the database host names
to the new primary database nodes VIP host names and port.

4. Update the EBS Database Source.Change directory to $EBS_DOMAIN_HOME/config/jdbc/ and

edit the EBSDataSource-<n>-jdbc.xml file as follows:
<url>jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS_LIST=(LOAD_BALANCE=YES)(FAILOVER=YES)(ADDRESS=(PROTOCOL=tcp)
(HOST=<SCAN HOST>)(PORT=<SCAN PORT>)))(CONNECT_DATA=(SERVICE_NAME=<service name)))</url>

5. Run AutoConfig on the run file system on all of the nodes.

$ $INST_TOP/admin/scripts/adautocfg.sh

6. To sync the patch file system with all of the changes, start by checking whether an
ADOP cycle is open.
$ adop -status

o If the ADOP cycle is open then execute AutoConfig on patch file system. (Skip
this step if ADOP cycle is not open)
$ $INST_TOP/admin/scripts/adautocfg.sh

o If there is no open ADOP cycle, source the patch file system - change
to $AD_TOP/bin/ and run the following command:.
 $ perl adconfig.pl contextfile= <INST_TOP>/appl/admin/PATCH CONTEXT.xml -syncctx

Next abort any incomplete online patching cycle, if exist, that was initiated on
primary site by executing the following command. Then it will allow to start new
adop cycle.
$ adop phase=abort

7. Start the services on the slave nodes:

$ adstrtal.sh <apps_user>/<apps_password>

6.4.2.3 After a Failover Remove the Primary Slave Nodes from the Configuration.

Following a failover to the standby site, the primary slave nodes no longer exist in the new
configuration and therefore it is essential that they are deleted using the following
command from the admin server node:
$ perl <AD_TOP>/patch/115/bin/adProvisionEBS.pl ebs-delete-node -contextfile=<CONTEXT_FILE>
-hostname=<node_name_to_be_deleted> -logfile=<LOG_FILE>

For further information on deleting a node, refer to My Oracle Support Knowledge Document
1383621.1, Cloning Oracle E-Business Suite Release 12.2 with Rapid Clone.

Section 7: Maintaining Oracle E-Business Suite with a Standby Database

This section describes how to apply an Oracle E-Business Suite patch on the primary and
incrementally update the standby database.

7.1 Prepare to Apply an Application Patch

7.2 Patch the Primary
7.3 After Applying the Patch

Note: If your standby database is running during a patch application, the database changes on
primary will be automatically pushed or replicated on the standby. To protect both the integrity
of the primary and standby from any problems during patch application, backup both databases
and Oracle E-Business Suite file system before patching. You may be able to achieve this more
quickly with certain types of disk systems that use mirroring.

7.1 Prepare to Apply an Application Patch

7.1.1 Stop/remove the recovery delay if it is set on the standby

If you have recovery delay set for redo log application on the standby, stop the delay. Stop the
media recovery before changing the delay option and start the media recovery after changing
delay option. On the standby database, run the following commands as a privileged user:
SQL>alter database recover managed standby database cancel;
SQL>alter database recover managed standby database nodelay;
SQL>alter database recover managed standby database disconnect from session;
7.1.2 Switch redo logs on primary database

On the primary database server, log into SQL*Plus as sysdba and issue the following commands
to switch logs; next discover the last log sequence number:
SQL>alter system switch logfile;
SQL>select sequence# -1 from v$log where status = 'CURRENT';

Note For Oracle RAC Configurations: Repeat this command to switch the logs on each
instance.

7.1.3 Ensure all log files are applied on the standby

On the standby database server, connect as sysdba and monitor the system to ensure that all log
files have been shipped and applied:
SQL>select sequence#, applied, to_char(first_time, 'mm/dd/yy hh24:mi:ss') first
from v$archived_log
order by first_time;

7.2 Patch the Primary

Open a patch cycle and apply the patch, either as a hotpatch or an online patch.

Note: If using hotpatch mode, stop the recovery on the standby. When using online patching
ensure that all the redo log files have been shipped and applied on the standby before starting
the cutover phase. This will isolate it and maintain its integrity.

Use the following commands to stop the redo shipping and recovery on the standby before
initiating cutover when using online patching:

On the standby stop the recovery using:

SQL>alter database recover managed standby database cancel;

On the primary stop the redo shipping:

SQL>alter system set log_archive_dest_state_<n>=defer;

Note: If there is a failover during a patching cycle, follow the steps in Appendix F: Handing
Online Patching Following a Standby Failover Event in order to abort the patch cycle.

7.3 After Applying the Patch

7.3.1 Complete the patch cycle

It is essential that you complete the ADOP patching cycle: prepare, apply, cutover, cleanup.
When using an online patch or hotpatch, run fs_clone to make sure that both run and patch file
systems are at the same patch level.
7.3.2 Synchronize the Primary and Standby Application Tiers

To syncronize the application tiers follow the steps below:

7.3.2.1 Put the standby database into snapshot standby mode

1. Synchronize the primary and standby database by forcing a redo log

switch on the primary (In the case of RAC, on all instances):
SQL>alter system switch logfile;

2. Follow all of the steps [1-10] in Section 5.1.1 to open the standby
database in Snapshot Standby mode.
7.3.2.2 Run AutoConfig

Execute AutoConfig on the run file system on each of the standby application tiers.
7.3.2.3 Apply the patches to the standby using hotpatch mode using the nodatabaseportion option

Note: Ensure that you have closed the ADOP cycle on primary site before applying the patch on
standby site.

Apply all the patches have been applied to the primary (using online patching), perform the
following steps on the standby:

1. Flip the file system on standby application tiers to be the same as the
primary if required:
$ perl $AD_TOP/patch/115/bin/txkADOPCutOverPhaseCtrlScript.pl -action=ctxupdate
-contextfile=<full path to current run context file on standy> -patchcontextfile=<full path
to current patch file system context file on standby> -outdir=<full path to out directory>

Note: If the command above fails, create the TXK_CTRL_ctxupdate directory

in the outdir directory if it doesn't exist. Once the file system has been
changed, run the environment file so that it now points to the new run file
system.

2. Run AutoConfig on the run file system before starting adop session.

3. If necessary, copy the $PATCH_TOP to the other application tiers. Apply the
patches using hotpatch in the order that they were applied on the primary.
Use options=nodatabaseportion so that the they are applied without updating
the database:
 $ adop phase=apply patches=patch number apply_mode=hotpatch options=nodatabaseportion

4. If you have previously applied FMW Patches (for example FMW 10.1.2
Oracle Home patches) apply those patches to the current run file system
on the standby site.

5. Put the snapshot standby database into Physical Standby mode. Execute
the steps under 5.1.3.

7.3.3 Reconfigure the standby database file systems (optional)

If you had to synchronize the apps utilities on the database server in the previous step, you
should also reconfigure the database server. As the ORACLE user on the standby database server,
first stop the listener if it is running, then use the cloning toolkit to define the database tier
topology at the standby site:
$ lsnrctl stop <SID>
$ cd < RDBMS_ORACLE_HOME>/appsutil/clone/bin
$ perl adcfgclone.pl dbTechStack

Answer the questions when prompted.

The previous step starts the database listener for primary services. You now need to stop and re-
start it on the standby:
$ lsnrctl stop <SID>
$ lsnrctl start < standby service name>

Appendix A: Oracle Net Files

This appendix is divided into the following three sections:

A1: The Common Listener Configuration for both Oracle RAC and Non-RAC
A2: TNS Alias Requirements in a non-RAC Configuration

A3: TNS Alias Requirements in an Oracle RAC Configuration

The examples in this section use the following convention where SID is same on both the
primary and physical standby:

Primary Physical Standby

Oracle Net Service Name dg12 dg12s
SID dg12 dg12

A1: Common Listener Configuration for RAC and Non-RAC

Example <TNS_ADMIN>/listener_ifile.ora file

##################################################################
#
# Created to define net services to support a Oracle Data Guard physical
# standby environment.
#
##################################################################
Standby LISTENER.ORA file when server is running as standby

dg12s =
(ADDRESS_LIST =
(ADDRESS=(PROTOCOL=TCP)
(HOST= <standby DB host>)
(PORT= <same as production>)
)
)
SID_LIST_dg12s =
(SID_LIST =
(SID_DESC =
(ORACLE_HOME= ) (GLOBAL_NAME=dg12s)
(SID_NAME=dg12)
)
)

STARTUP_WAIT_TIME_dg12s = 0
CONNECT_TIMEOUT_dg12s = 10 TRACE_LEVEL_dg12s = OFF

LOG_DIRECTORY_dg12s = <same as production>

LOG_FILE_dg12s = STDBY
TRACE_DIRECTORY_dg12s = <same as production>
TRACE_FILE_dg12s = STDBY
ADMIN RESTRICTIONS_dg12s = OFF

A2: TNS Alias Requirements in a non-RAC Configuration

Example < TNS_ADMIN>/<CONTEXT_NAME>_ifile.ora file, used to support fal_client/server communication.

##################################################################
#
# Created to define net services to support a Oracle Data Guard physical
# standby environment.
#
##################################################################

#
# The Oracle Data Guard physical standby of the primary runs on the <standby host>.
# Oracle Data Guard uses the TCP protocol only.
#
# This entry must point to the current standby server - i.e. this is the
# fal_client alias used to communicate from primary to standby.
# MUST BE CHANGED on switchover:
dg12s=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)
(HOST= < standby DB host name>)
(PORT= <Same PORT as primary)
)
(CONNECT_DATA=(SID=dg12)
)
) #
# Fetch Archive Log (FAL) service definition.
# This entry can be set up for use when THIS server hosts a
# standby database (thus will not need to be changed on switchover),
# and should point to what would be the PRIMARY AT THAT TIME -
# i.e. this is the fal_server alias used to communicate from the standby to primary.
dg12=
(DESCRIPTION=
(ADDRESS=(PROTOCOL=tcp)
(HOST= <prod DB host name when this is standby>)
(PORT= <same port as primary)
(CONNECT_DATA=(SID=dg12) ) )

A3: TNS Alias Requirements in an Oracle RAC Configuration

For Oracle RAC configurations, the entries for the TNS aliases should be as follows, where
prod and stdby are the primary and standby service names, respectively. The entries must be the
same on all nodes in the cluster as well as in the standby instances.
A sample < TNS_ADMIN>/<CONTEXT_NAME>_ifile.ora file, used to support fal_client/server communication.
##################################################################
#
# Created to define net services to support a Oracle Data Guard physical
# standby environment.
#
##################################################################

#
# The Oracle Data Guard physical standby of primary runs on the <standby host>.
# Oracle Data Guard uses the TCP protocol only.
#
# This entry must point to the current standby server - i.e. this is the
# fal_client alias used to communicate from primary to standby.
# MUST BE CHANGED on switchover:
prod=
(DESCRIPTION=
(LOAD_BALANCE=NO)
(FAILOVER=YES)
(ADDRESS_LIST=
(ADDRESS=
(PROTOCOL=tcp)
(HOST=<Primary DB Host1>)
(PORT=<dbport>))
(ADDRESS=
(PROTOCOL=tcp)
(HOST=<Primary DB Host2)
(PORT=<dbport>)) )
(CONNECT_DATA=(SERVICE_name=prod))
)
stdby=
(DESCRIPTION=
(LOAD_BALANCE=NO)
(FAILOVER=YES)
(ADDRESS_LIST=
(ADDRESS=
(PROTOCOL=tcp)
(HOST=<Stndby DB Host1)
(PORT=<dbport>))
(ADDRESS=
(PROTOCOL=tcp)
(HOST=<Stndby DB Host2>)
(PORT=<dbport>))
)
(CONNECT_DATA=(SERVICE_name=stdby)(UR=A))
)

Appendix B: Using Oracle Applications Manager to Register a Standby Database

1. From a client, connect to OAM to register the standby database server as

a node. Navigate as follows:

Site Map > Administration > System Configuration > Hosts > Register
(button under Other Hosts).

2. Next, use OAM to add this host to the list of hosts that need access to the
database:

Applications Dashboard > Security > Manage Security Options > Enable
Restricted Access > Run Wizard.
3. Select the host you just added, and click 'Continue'.
4. If the displayed list is correct and includes your new host, click 'Submit'.

Appendix C: Example Standby Database Commands

Example of how to add Redo Log Files on the Standby

The following commands for example can be used on a non-RAC database to add standby redo log file.

SQL> Alter database add standby logfile group 4 ('/d1/MAABCU/primary/dg12data/log5.dbf') size 50M ;
SQL> Alter database add standby logfile group 5 ('/d1/MAABCU/primary/dg12data/log6.dbf') size 50M ;
SQL> Alter database add standby logfile group 6 ('/d1/MAABCU/primary/dg12data/log7.dbf') size 50M ;

Note For Oracle RAC Configurations: Add standby redo log files for each thread using the
following command as an example which uses a 300MB file:
SQL> alter database add standby logfile thread 1 group 10 ('+DATA/dbfiles/stdbylog1.dbf') size 300M;

Gathering temporary tablespace information to create a temporary tablespace on the standby

If RMAN is used to create the standby database, it creates the temp files in the standby database.
If RMAN is not used, the temp files has to be manually created after the creation of standby
database. For that reason, it is required to note down the temp files informantion using the
following command.
SQL>select file_name, bytes from dba_temp_files;
FILE_NAME BYTES
------------------------------------------
/d1/MAABCU/primary/dg12data/tmp1.dbf 2097152000

Appendix D: Using RMAN to Create a Physical Standby Database

Perform the following steps when using RMAN to create a physical standby database. It has
following two section.
Appendix D1: Oracle non-RAC RMAN Clone Example
Appendix D2: Oracle RAC RMAN Clone Example

Appendix D1:Oracle non-RAC RMAN Clone Example

1. Ensure that you have completed steps 4.1 to 4.4 in Section 4.

2. Ensure that you have completed the network configuration and started the
standby database listeners.

3. Start the standby database in nomount mode:

$ sqlplus / as sysdba
SQL>startup nomount;

4. Verify the connections to primary and standby database.

$ sqlplus sys/<password>@<TNS alias to primary>
$ sqlplus sys/<password>@<TNS alias to standby> as sysdba

5. Create an initialization parameter file for the standby by running the

following command on the primary:
SQL>create pfile from spfile
 6. Copy the init_<SID>.ora from the primary to the standby.

7. Change the db_unique_name to the standby db_unique_name; this should be

different from primary.

For example: db_unique_name=dg12s

8. For log_archive_dest_2, specify the primary db_unique_name.

For example, log_archive_dest_2='SERVICE=dg12s LGWR ASYNC

DB_UNIQUE_NAME=dg12'
db_unique_nameshould specify the destination to ship redo logs to, from the
standby site to the primary site after switch over.
9. Connect as sysdba and issue the following command to start up but not
mount the standby:
SQL>startup nomount pfile= <pfile created in above step>

10. Connect to target and auxiliary databases using RMAN:

$ rman target sys/manager@dg12 auxiliary sys/manager@dg12s
(In this example, dg12 = primary service name and dg12s = standby service name)
Recovery Manager: Release 11.2.0.1.0 - Production on Wed Jan 20 03:16:56 2010 Copyright (c)
1982, 2009, Oracle and/or its affiliates. All rights reserved.
connected to target database: dg12 (DBID=3753412759)
connected to auxiliary database: dg12 (not mounted)
RMAN>

11. Execute the RMAN DUPLICATE command on standby:

RMAN>DUPLICATE TARGET DATABASE FOR STANDBY FROM ACTIVE DATABASE
DORECOVER
SPFILE SET "db_unique_name"="dg12s"
SET LOG_ARCHIVE_DEST_2="service=dg12s ASYNC REGISTER
VALID_FOR=(online_logfile,primary_role)DB_UNIQUE_NAME=dg12"
SET FAL_SERVER="dg12" COMMENT "Is primary"
SET DIAGNOSTIC_DEST=$ORACLE_HOME/admin/<STNDBY_CONTEXT_NAME> COMMENT "DIAGNOSTIC
Destination on standby server"
SET UTL_FILE_DIR= <Appropriate value depending on standby context name >
NOFILENAMECHECK;

In the above example, RMAN automatically copies the server parameter

file to the standby host and then starts the auxiliary instance using this
file.

Appendix D2: Oracle RAC RMAN Clone Example

Copy the RDBMS_ORACLE_HOME from primary nodes to the corresponding standby nodes. After the
database software copies are complete, log into the standby database server as the oracle user
and execute the following commands to update the file system configurations for the new
environment. Since your environment scripts are not yet set up, you will need to manually
resolve the reference to<RDBMS_ORACLE_HOME>.
 1. Run adclonectx and then adcfgclone dbTechstack on all nodes.
$ perl <RDBMS_ORACLE_HOME>/appsutil/clone/bin/adclonectx.pl \
contextfile=<RDBMS_ORACLE_HOME>/appsutil/<SID>_<Primary Application Host>.xml \
template=$Standby_ORACLE_HOME/appsutil/template/adxdbctx.tmp \
pairsfile=<RDBMS_ORACLE_HOME>/appsutil/clone/pairsfile.txt
$ perl <RDBMS_ORACLE_HOME>/appsutil/clone/bin/adcfgclone
dbTechstack <STANDBY_RDBMS_ORACLE_HOME>/appsutil/<SID>_<NODE_NAME>.xml

Note:

o should contain the instance specific values for

pairsfile.txt
parameters s_undo_tablespace, s_dbClusterInst and s_db_oh.
o Answer the questions when prompted during adclonectx.

o When asked Current node is the first node in an N Node RAC Cluster
(y/n)[n]:y enter 'Y' for each node otherwise it will prompt for
"Live RAC node", which will not be available at this time.
Alternatively, this could be avoided by -initialnode argument to
the adclonectx.pl for each nodes.
o When asked Do you want the inputs to be validated (y/n) [n] ? :
[n] enter 'N' for each node as the database that it needs to connect
to is not available.
o Modify the thread parameter in the init parameter file as the thread
parameter must be unique for each instance.

2. Use the RMAN 'duplicate database' command to create and mount the
standby database over the network. On the first node of the standby
system, start up the database in nomount mode (using a pfile) and run the
following command:
$ rman target sys/manager@prod auxiliary /
Recovery Manager: Release 11.1.0.7.0 - Production on Thu Mar 11 02:17:43 2010
Copyright (c) 1982, 2007, Oracle. All rights reserved.
connected to target database: PROD (DBID=3908691352)
connected to auxiliary database: PROD (not mounted)
RMAN> DUPLICATE TARGET DATABASE FOR STANDBY FROM ACTIVE DATABASE;

3. Enable recovery on the node that is to be used for the recovery process.

Appendix E: Using Oracle Data Guard Broker (DGMGRL) to Manage Standby Databases

Oracle Data Guard broker is an easy to use interface to manage standby databases. It is easy to
perform role transitions with a single command either for switchover or failover. This section
covers DGMGRL - the command line interface used to manage standby databases. This
appendix is devided into the following sections:

E1. Prerequisites
E2. Configure Oracle Data Guard Broker
E3. Role Transitions

E1. Prerequisites

Prior to using Data Guard broker, the standby database needs to be configured.

You must be using a server parameter file (spfile).

The Data Guard broker starts database instances during switchover or
failover using a statically registered service name. Therefore, it is
necessary to add a static descriptor to the custom listener file:
[<TNS_ADMIN>/<CONTEXT_NAME>_ifile.ora].

If you choose the DGMGRL default, then configure as per option 1

below; if you are using a different static descriptor and then set the
DGMGRL StaticConnectIdentifier property, as per option 2.

o Option 1:
The default option is for the broker to assume the
service <db_unique_name>_DGMGRL.<db_domain> has been statically
registered with the listener of each instance. Add an SID_DESC entry
as seen below:
(SID_DESC =
(GLOBAL_DBNAME=<DB_UNIQUE_NAME>_DGMGRL.us.oracle.com)
(ORACLE_HOME= <ORACLE_HOME>)
(SID_NAME = <SID>)
)

o Option 2:
Set the StaticConnectIdentifier property of both the primary and
standby databases to a TNS alias that resolves to a statically
registered descriptor.
DGMGRL>edit database <Primary Database> set property
StaticConnectIdentifier='<dg_prim>' -- where dg_prim is TNS alias to connect the
Primary
DGMGRL>edit database <Standby Database> set property
StaticConnectIdentifier='<dg_stndby>' -- where dg_stndby is TNS alias to connect
the Standby

Add the two TNS aliases to ifile (<sid>_<node>_ifile.ora)

under TNS_ADMIN on both standby and primary.

For example:
dg_prim=(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<primary host>(PORT=<port>))
(CONNECT_DATA=(SID=<sid>)(SERVER=DEDICATED)))
dg_stndby=(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=<standby host>(PORT=<port>))
(CONNECT_DATA=(SID=<sid>)(SERVER=DEDICATED)))

E2. Configure Oracle Data Guard Broker

 1. Start the Data Guard broker on both the primary and standby databases.
The Data Guard broker will create a configuration and backup file in the
location specified by the initialization parameterDG_BROKER_CONFIG_FILEn. The
default location is $ORACLE_HOME/dbs directory.
SQL>alter system set dg_broker_start=TRUE.

2. Configure the Data Guard broker using DGMGRL.

#dgmgrl sys/****@<primary database alias>
DGMGRL>CREATE CONFIGURATION '<Any Name>' AS PRIMARY DATABASE IS '<db_unique_name>' CONNECT
IDENTIFIER IS <Primary database alias>;

3. Add the standby database using following command:

SQL>ADD DATABASE '<standby unique name>' AS CONNECT IDENTIFIER IS <Standby TNS Alias>;

4. View the configuration using the Show Configuration command.

5. Set the following Data Guard broker properties:

1. Set the configuration protection mode to maximum availability. At

any time, you can change the protection mode of the
configuration. Note that this protection mode requires that there
be at least one standby database configured to receive the standby
redo log files, with its LogXptMode configurable database
property set to SYNC on both the primary and standby.
DGMGRL>EDIT database <database name> set property LogXptMode='SYNC'
DGMGRL>EDIT CONFIGURATION SET PROTECTION MODE AS MAXAVAILABILITY;

2. Do not enable FAST_START_FAILOVER as automatic failover is not

supported.

E3. Role Transitions

E3.1 Switchover

1. Verify that the primary and the target standby databases are in the
following states: primary TRANSPORT-ON and standby APPLY-ON.

On the Primary On the Standby

DGMGRL> show database stndby
DGMGRL> show database dbbrok
Database - stndby
Database - dbbrok
Role: PHYSICAL STANDBY
Role: PRIMARY
Intended State: APPLY-ON
Intended State: TRANSPORT-ON
Instance(s): dbbrok
Instance(s): dbbrok
Database Status: SUCCESS
Database Status: SUCCESS

2.
 3. Issue the switchover command:
DGMGRL>switchover to <standby database>;

4. Verify the switchover has been successful:

DGMGRL>show configuration
Databases:
stndby - Primary database
dbbrok - Physical standby database

5. To complete the switchover, follow the steps in Section 6 starting at

step 6.1.6 Complete database configurations.

E3.2 Switchback

Follow the same steps as in the switchover section, but change the database name to the name of
the standby after switchover as in the following example:
DGMGRL> switchover to dbbrok; --> where dbbrok is the name of the current standby after a switchover.

E3.3 Failover

There are two types of failover using Oracle Data Guard broker: manual failover and automatic
failover. Note that automatic failover, which uses the fast start failover option, is not supported
for Oracle E-Business Suite.

Manual Failover
1. Connect DGMGRL to the standby database.
dgmgrl sys/manager@<Standby Database alias>
DGMGRL> failover to <Standby Database>
Performing failover NOW, please wait...
Failover succeeded, new primary is "stndby"

2. To complete the failover, follow the steps in Section 6 starting at

step 6.2.7 Complete the database configuration.

Automatic Failover

Automatic failover is not currently supported with Oracle E-Business Suite, as you need to run
AutoConfig before starting the standby. This section will be updated when automatic failover is
certified.

Appendix F: Handling Online Patching Following a Standby Failover Event

Oracle E-Business Suite Release 12.2 supports online patching. However, if there is a failover
event during a patch cycle, there is no option to resume the patch on the standby. If you want to
apply the patches to the standby, you will first need to run a script to modify the status of the
online patching and then abort the current patch cycle. Steps are included for both possible
standby application tier configurations.
For Configuration 1: Standby Application Tiers Connected to the Standby Database

The steps to perform on a failover during patching are as follows:

1. Make sure that you have applied all the patches listed in PreRequisite
Patches in section-2.

2. Before running fnd_conc_clone.setup_clean in step 3, run the following

query to get the appltop_id for each of the primary nodes:

SQL>SELECT aat.appl_top_id,node_name
FROM FND_OAM_CONTEXT_FILES focf, AD_APPL_TOPS aat
WHERE aat.name=EXTRACTVALUE(XMLType(focf.TEXT),'//APPL_TOP_NAME')
AND focf.node_name=<Primary Node Name>
AND EXTRACTVALUE(XMLType(TEXT),'//file_edition_type') = 'run';

3. Execute the instructions in 6.2.7 Complete the database configuration and

all steps in 6.4 Configure the Application Tier After Role Transition
(Switchover/Failover/Switchback).

4. Get the appl_top_id for each of the standby nodes using the query in step 2
by substituting the <primary node name> with the <standby node name> .

5. In order to be able to restart a patching cycle, you need to change the

node names from the primary to the standby. As the APPS user, execute
the following:
SQL>exec AD_ZD_FIXER.fix_adop_repo_tables(<primary appltop_id (from Step 2)>,<primary node
name>, <Standby appltop_id (from step 4>,<standby node name>);

6. The EBS Technology Codelevel Check needs to be run on the database

node. Download and apply Patch 17537119. Execute checkDBpatch.sh on
database node.

Note: If you do not complete this step, you will be prompted to do so

when you run prepare.

7. On the standby, abort online patching that was initiated on primary site by
executing the following command:

$ ADOP phase=abort
You will then be able to restart the patch cycle.

For Configuration 2: Standby Application Tiers Connected to the Primary Database

1. Perform the failover steps detailed in 6.2 Perform a Failover to configure

the application tiers.

Appendix G: Creating a Non-RAC Standby for a RAC Primary

G1. Configure the Primary RAC to Create a Non-RAC Standby

G2. Create the Physical Standby

G1. Configure the Primary RAC to Create a Non-RAC Standby

Follow the instructions given in Section 3: Preparing the Primary Database for Standby
Database Creation to configure the primary RAC system.

G2. Create the Physical Standby

This section describes how to create the physical standby.

1. Copy the ORACLE_HOME file system to the standby database server.

Copy the ORACLE_HOME file system to the standby database server. If you
natively compile your PL/SQL, be sure to include the file system
directories containing the compiled objects. The standard location for this
is RDBMS ORACLE_HOME/plsql/nativelib.

2. Backup the primary RAC using RMAN.

You should take the backup of the primary RAC database and then copy
the backupsets to the standby server. Refer to step 1 of Appendix D: RAC
RMAN Clone Example for complete RMAN instructions.
3. Perform file-based configuration on the standby database server.

Once the database software copies have completed (which may happen
before the database itself has finished copying), log into the standby
database server as the ORACLE user and execute the following commands to
update the file system configuration for the new environment. Since your
environment scripts are not yet set up, you will need to manually resolve
the reference to RDBMS ORACLE_HOME.
$ cd RDBMS ORACLE_HOME/appsutil/clone/bin
$ perl adcfgclone.pl dbTechStack
 Answer the questions when prompted. If you receive any errors
registering the new ORACLE_HOME, follow the instructions given by the script
to correct them.

For the question: Target instance is a Real Application Cluster (RAC) instance
(y/n) [y]: n

Enter 'n' as the standby is non-RAC.

Note: Relink oracle with the rac_off option. If adlnkoh.sh fails then
run adcfgclone again.

Your ORACLE user environment scripts are now ready to use. Source them
prior to running any commands in the next steps, using the appropriate
OS command. For example, in sh or ksh on UNIX:

$ cd RDBMS ORACLE_HOME
$ ./STNDBY_CONTEXT.env

If you have implemented native PL/SQL compilation, set up an rsync job

from the primary database server to the standby database server for the
file system directories containing the compiled objects. The standard
location for this is RDBMS ORACLE_HOME/plsql/nativelib.

Modify the initialization parameter as per 4.6 Modify the Database

init.ora Parameters on the standby server.

4. Stop the standby listener and configure it for net redo transmission

Stop the listener and modify the <StandbySID>_ifile.ora to configure net

redo transmission. For further information refer to Section 4: Creating a
Physical Standby Database.

5. Startup the instance in nomount:

$ sqlplus / as sysdba
SQL>startup nomount

6. Create the standby database using RMAN:

rman target sys/<passwd>@<primaryservice> auxiliary sys/<passwd>@<stndbyservice>
rman>DUPLICATE TARGET DATABASE FOR STANDBY FROM ACTIVE DATABASE;

When the command above has executed, the database will be in the
 mount state having used the (pfile) initialization parameter file.

Put the standby database in to managed recovery mode:

SQL>alter database recover managed standby database disconnect from session;

For the applications tier configuration, follow the steps given in Section
4: Creating a Physical Standby Database. Start with the steps from 4.8
Start shipping redo from the primary to the standby database server and
complete all of the steps to the end of that section. Next perform all of the
steps in Section 5: Configuration on Application Tiers After Standby
Database is Enabled.

Change Log

Date Description
17-May-2016 Added Note section in section 5.1.2 to include 'dualfs' option in adcfgclone.
Updated applicable version(s) in section 2.2 to include Oracle E-Business Suite
09-Feb-2016
12.2.5
23-Dec-2015 Added appendix G for the scaledown scenario.
19-Nov-2015 Update Prerequisite patch in Section-2 to include Delta 7.
04-Nov-2015 Updated for 12.1.0.2 RAC Database with Multiple Applications Tiers.
07-Aug-2015 Performed editorial review.
17-Jul-2015 Updated section 5 with Snapshot database approach.
11-Mar-2015 Initial publication.

My Oracle Support Knowledge Document 1963472.1 by Oracle E-Business Suite Development

Copyright 2015, Oracle and/or its affiliates. All rights reserved.

Didn't find what you are looking for?