An Oracle White Paper

January, 2012

Near Zero Downtime Upgrade for Oracle

Communications Billing and Revenue
Management Using Oracle GoldenGate
 Near Zero Downtime Upgrade for Oracle Communications Billing and Revenue Management Using Oracle GoldenGate

Executive Overview ........................................................................... 1

Introduction ....................................................................................... 1
A Robust Solution for Near Zero Downtime Oracle
Communications BRM Upgrades....................................................... 3
Oracle GoldenGate Application Architecture...................................... 4
Capture ......................................................................................... 4
Trail Files ....................................................................................... 5
Delivery ......................................................................................... 5
Key Features ................................................................................. 5
Deployment Models for Near Zero Downtime Oracle
Communications BRM Upgrades....................................................... 6
Unidirectional Deployment Model .................................................. 6
Failback Deployment Model .......................................................... 7
Implementing a Near Zero Downtime Oracle
Communications BRM Upgrade Solution........................................... 8
Scope Phase ................................................................................. 9
Develop Phase .............................................................................. 9
Test Phase .................................................................................. 10
Roll-out Phase ............................................................................. 11
Ongoing Data Analysis ................................................................ 12
Conclusion ...................................................................................... 13
 Near Zero Downtime Upgrade for Oracle Communications Billing and Revenue Management Using Oracle GoldenGate

Executive Overview

Performing a near zero downtime upgrade of Oracle Communications Billing and Revenue
Management (BRM) application is possible using Oracle GoldenGate. Alleviating downtime
during Oracle Communications BRM upgrades mitigates disruption of business operations and
minimizes the potential for revenue leakage for prepaid customers.

Introduction

Oracle Communications Billing and Revenue Management is an end-to-end revenue

management system for communications and media service providers. Many of the world's
largest and most innovative service providers—including France Telecom, Vodafone,
Swisscom, Savvis, Sirius XM Satellite Radio and Nintendo —rely on Oracle Communications
Billing and Revenue Management to achieve their most critical business goals.

The upgrade process for enterprise software applications like Oracle Communications BRM
can be quite complex and tedious. Until now there were only two approaches to upgrading
Oracle Communications BRM systems: in-place upgrades and the parallel system run.

The in-place upgrade approach consisted of simply shutting down the current Oracle
Communications BRM system, upgrading it to the latest release of Oracle Communications
BRM, and then bringing the system back online. This “Big Bang” approach required extensive
downtime to complete and put heavy risk on the business.

The parallel system run approach attempts to have two Oracle Communications BRM systems
running live in parallel. Unfortunately, this approach could only keep a portion of the data in-
sync between the two systems as not all data feeds could be applied to both databases.

1
 Near Zero Downtime Upgrade for Oracle Communications Billing and Revenue Management Using Oracle GoldenGate

There is now a third and better approach that uses Oracle GoldenGate to upgrade Oracle
Communications BRM systems with near zero downtime. Oracle GoldenGate significantly
mitigates the risk to the business during the upgrade process with the ability to stage a full
copy of the new Oracle Communications BRM system on the latest software release without
impacting the current BRM system. The purpose of this white paper is to provide an overview
of the new near zero downtime approach of upgrading Oracle Communications BRM systems.

2
 Near Zero Downtime Upgrade for Oracle Communications Billing and Revenue Management Using Oracle GoldenGate

A Robust Solution for Near Zero Downtime Oracle Communications

BRM Upgrades
Oracle GoldenGate enables near zero downtime upgrades for mission-critical Oracle Communications
BRM implementations. The solution allows companies to maximize revenue-generating activities with
the least amount of impact to business users by significantly reducing the downtime required while
performing a BRM upgrade. Designed for reliability and flexibility, Oracle GoldenGate is:
 Noninvasive. The software uses a nonintrusive, low-overhead capture process that reads
changed data from database transaction logs and does not burden the source system.
 Fast. As changes are recorded, they are sent in real time to the target system that is set up and
tested for production. Changed data can be transformed for upgrade “in flight” and is
continuously applied until the customer switches over to the new application.
 Deployable in multiple models. Oracle GoldenGate components are loosely coupled and
preserve transaction integrity across a complete set of topologies: unidirectional, with data
flowing one way; and failback, which provides the ability to failback users to the instance with
the prior BRM release.

Oracle GoldenGate’s unique software platform does not require any changes to the source Oracle
Communications BRM system and uses minimal system resources. The solution supports Oracle
Communications BRM clients running Oracle Communications BRM 7.2 or higher.

Its asynchronous, decoupled architecture allows for synchronization of source and target systems
running different Oracle Communications BRM releases.

Deploying Oracle GoldenGate for near zero downtime Oracle Communications BRM upgrades
enables businesses to:

 Maximize revenue-generating activities. By upgrading with minimized downtime,

organizations can continue revenue-generating business operations with little interruption.
 Reduce implementation time and cost. By removing the need to make invasive changes to
the source and target systems and by seamlessly moving data in real time, upgrades can be
implemented in a dramatically reduced timeframe.
 Leverage new Oracle Communications BRM features. By significantly reducing
downtime, the most critical upgrade hurdle, IT organizations can migrate users to new releases
of Oracle Communications BRM to take advantage of more up-to-date features.
 Remove the barriers of a single vendor operating platform. Oracle GoldenGate is
heterogeneous. Oracle GoldenGate is available for all supported Oracle Communications
BRM database server operating systems. For instance, the source database server operating
system could be Solaris and the target operating system could be Linux. It also provides the
option to migrate to different operating systems and database platforms. A database upgrade
or migration can be included in the Oracle Communications BRM upgrade process. As with

3
 Near Zero Downtime Upgrade for Oracle Communications Billing and Revenue Management Using Oracle GoldenGate

the application upgrade, from the perspective of the business user, there is little downtime or
penalty for making such a move.

Oracle GoldenGate Application Architecture

Oracle GoldenGate moves high volumes of changed data between heterogeneous databases with sub-
second latency while preserving transaction integrity. Leveraging decoupled components, the software
platform can be configured to enable a variety of solutions for continuous availability and disaster
tolerance as well as real-time data integration. The application is designed to allow each component to
perform its tasks independently of the others, which eliminates bottlenecks, optimizes data transfer,
and maximizes reliability.

Oracle GoldenGate consists of three modules: Capture, Trail Files, and Delivery (See Figure 1).

Figure 1. Oracle GoldenGate leverages a decoupled architecture to optimize real-time data capture and transfer.

Capture
The Capture module resides on the source database and looks for new transactional activity. Capture
reads the result of insert, update, and delete operations by directly accessing the database’s transaction
(redo) logs, and then immediately captures that new data for distribution. The Capture module
supports a wide range of database versions, including Oracle Database, Microsoft SQL Server, IBM
DB2 mainframe and LUW, Ingres, Sybase, Enscribe, SQL/MP, SQL/MX, and Teradata running on
Linux, UNIX, Windows, z/OS, and HP NonStop platforms.

4
 Near Zero Downtime Upgrade for Oracle Communications Billing and Revenue Management Using Oracle GoldenGate

To reduce infrastructure load and eliminate data inconsistencies, the Capture module moves only
committed transactions (intermediate activities and rolled-back operations are filtered out). Further
optimization can be achieved through transaction grouping and compression.
A second Capture component, called a Pump, can be used to move Trail Files from the source system
to multiple target systems. This configuration enhances the fault tolerance and reliability of the overall
Oracle GoldenGate environment. In the event of a network failure, Oracle GoldenGate can continue
to capture transactions because the data can be queued up locally in the Trail Files on the source,
enhancing the recoverability in case of database failures.

Trail Files
Trail Files are part of the Oracle GoldenGate queuing mechanism and store the captured changed data
in a transportable, platform-independent universal data format. Trail Files reside on the source and
target servers, but exist outside the databases to ensure heterogeneity, improved reliability, and minimal
data loss. This architecture minimizes any impact on the source system because no additional tables or
multiple queries to the database are required to support the capture processes. The Capture module
reads once, then immediately moves the captured data to the external Trail Files for delivery to the
target(s).
If there is an outage at the source or the target, Trail Files contain the most recently changed data up to
the point of the outage, which are applied once the systems are back online.

Delivery
The Delivery module takes the data transactions from the latest Trail File and applies that data to the
target using the native SQL for the relevant database management system—delivery can be made to
any Open Database Connectivity compliant database. The Delivery module applies each transaction in
the same order as it was committed and within the same transactional context as at the source, to
ensure consistency and referential integrity at the target. Delivery uses a number of techniques to
optimize the application of data to the target, and changed data can be provided as a flat file to
integrate with third-party extract, transform, and load products. Oracle GoldenGate can format text in
any way, including but not limited to XML and delimited format to be published to enterprise
messaging systems.

Key Features
The following Oracle GoldenGate features facilitate and streamline the upgrade process:

 Transformations and mappings. Oracle GoldenGate can flexibly accommodate transformations

and mappings within either its Capture or Delivery modules, without requiring a middle-tier
server. The application supports table and row filtering based on user-defined criteria, and explicit
mapping and transformation rules can be applied via native functions, user-supplied code, and
stored procedures. These rules can range from simple column assignments to more-complex

5
 Near Zero Downtime Upgrade for Oracle Communications Billing and Revenue Management Using Oracle GoldenGate

transformations, for which the application provides a suite of date, math, string, and utility
functions.
 Flexible topology support. Oracle GoldenGate’s architecture allows customers to support a
variety of topologies, including one source to one target, one-to-many, many-to-one, many-to-
many, cascading, and bidirectional configurations.
 Routing and compression. Oracle GoldenGate uses TCP/IP for sending data, so no
geographical distance constraints are imposed between the source and target systems. An
additional compression process can be applied to the data as it is routed.
 Data encryption. Oracle GoldenGate offers encryption to ensure secure, confidential data
transmissions both domestically and internationally.
 Conflict detection and resolution. Bidirectional implementations with more than one active
database require conflict detection and resolution capabilities, because both systems are actively
processing and sharing database transactions. Oracle GoldenGate provides conflict detection and
resolution options that can be implemented globally, object by object, based on data values,
complex filters, and event-driven criteria.

Deployment Models for Near Zero Downtime Oracle

Communications BRM Upgrades
Oracle GoldenGate uses two well-established deployment models for Oracle Communications BRM
near zero downtime upgrades:

 Unidirectional. Changed data flows in one direction with a static failback environment.
 Failback. Changed data flows in one direction with a dynamic failback environment.

Unidirectional Deployment Model

For a unidirectional deployment, changed data flows in one direction and is transformed for the new
Oracle Communications BRM release in flight. To accomplish this, the Oracle GoldenGate Capture
module is enabled on the source (prior release of Oracle Communications BRM) to capture any
changes as they are made in the production Oracle Communications BRM database. A parallel, target
environment is established with a copy of the source database, which is then upgraded to the new
Oracle Communications BRM release. During the whole upgrade process, Oracle GoldenGate
captures new transactions and stores them in the Trail Files.
Once the Oracle Communications BRM upgrade is complete, the changes captured in the source
database are sent to the target and upgraded in-flight using the pre-configured “upgrade logic”, which
is dependent upon the source and target Oracle Communications BRM releases. The upgrade logic
transforms the transactions based upon the database schema differences between the source Oracle

6
 Near Zero Downtime Upgrade for Oracle Communications Billing and Revenue Management Using Oracle GoldenGate

Communications BRM release and the new target Oracle Communications BRM release. Once the
sourced and target databases are synchronized in real time, users can be switched over to the new
release (Figure 2).

Figure 2. In a unidirectional deployment model, data flows in one direction from the source to the target.

The unidirectional model, does not provide a failback option for the old Oracle Communications BRM
system. The old Oracle Communications BRM system is left intact and remains at a specific point in
time, until the business decides the risk of removing the old system is zero. If the customer is forced to
cease using the new Oracle Communications BRM release, the prior system is available, but the data
would be current as of the cut-over time. Any subsequent transactions processed in the new Oracle
Communications BRM release would not be present.

Failback Deployment Model

For customers that want the ability to failback and retain any changed data, the option is available to
deploy a failback deployment model. This model is based on the unidirectional model but extended to
provide customers the ability to switch back from the new Oracle Communications BRM release to the
prior release without data loss in a worst-case scenario. This would be accomplished by implementing
the Oracle GoldenGate processes in reverse to flow changed data from the new Oracle
Communications BRM release back to the prior Oracle Communications BRM release and
“downgrading” the changes in-flight. Downgrading is the process by which the transactions of the new
Oracle Communications BRM release are transformed to the prior Oracle Communications BRM
release.
This failback model enables customers to go back to the prior release of Oracle Communications BRM
immediately and without data loss. Under this model, source and target environments cannot be
both active at the same time. Only one Oracle Communications BRM system can be active at any
given time (Figure 3).

7
 Near Zero Downtime Upgrade for Oracle Communications Billing and Revenue Management Using Oracle GoldenGate

Figure 3. In a failback deployment model, data is captured on the target and replicated on the source to provide a
backup instance.

Implementing a Near Zero Downtime Oracle Communications BRM

Upgrade Solution
Deploying Oracle GoldenGate for near zero downtime Oracle Communications BRM upgrades
requires a specific configuration for each deployment model (unidirectional or failback). The
implementation of the solution follows a standard methodology consisting of four phases: scope,
develop, test, and roll-out (Figure 5).

Figure 5. Oracle GoldenGate supports a robust and flexible methodology for near zero downtime upgrades.

Most implementations follow a typical systems development lifecycle. To aid you in developing a
complete picture of the implementation process for Oracle GoldenGate in a near zero downtime

8
 Near Zero Downtime Upgrade for Oracle Communications Billing and Revenue Management Using Oracle GoldenGate

Oracle Communications BRM upgrade environment, the following subsections describe the types of
tasks associated with each phase of the above methodology, as well as best practice recommendations.

Scope Phase
In the scope phase, you set the project boundaries. The goal of this phase is to capture as much detail
as possible for all the individual components of the solution and to determine the most appropriate
deployment model to meet the business requirements.

Requirements Gathering
The following list summarizes key information that must be collected from each environment.
 Source (prior release of Oracle Communications BRM)
o Physical location
o Relational database management system (RDBMS) vendor and version
o RDBMS host operating system, version, hardware vendor, and number of CPUs or
cores
o Identify custom objects in source Oracle Communications BRM database to be
replicated to the target (new release of Oracle Communications BRM)
o Note: Identify volume of changes because these could affect future performance tuning
 Target (new release of Oracle Communications BRM)
o Physical location
o Note: If source and target are remote, determine whether network will support real-time replication
o RDBMS vendor and version
o Note: Determine need for a database upgrade or migration at this time
o RDBMS host operating system, version, hardware vendor, and number of CPUs or
cores
o See if Oracle Communications BRM customizations can be replaced by new BRM
OOB Features
o Merge Oracle Communications BRM C customizations

Develop Phase
Using the information gathered in the scope phase, the develop phase is composed of 2 main tasks:

 Oracle Communications BRM Upgrade and Downgrade Logic. Identify the current
Oracle Communications BRM release and the new Oracle Communications BRM release that
the system will be upgraded to. This information is used to determine which Oracle
GoldenGate pre-created upgrade and downgrade logic configuration files must be used to
perform the near zero downtime upgrade.
 Target (new Oracle Communications BRM release) initialization method. There are
many different methods for instantiating a target environment, but the goal is always the same:
to establish a copy of the existing production Oracle Communications BRM database in the
target Oracle Communications BRM system for a known date and time.

9
 Near Zero Downtime Upgrade for Oracle Communications Billing and Revenue Management Using Oracle GoldenGate

Target Initialization

There are several methods and tools for instantiating the target Oracle Communications BRM
database, including bulk data export utilities; bulk data import utilities; disk mirroring, where a mirrored
set of drives is mounted on the target database server; and simple backup and restore procedures. For
Oracle Databases, Oracle’s RMAN backup and recovery, Physical Data Guard, and Transportable
Tablespaces features can be used as well.

Test Phase
It is critical that the new release of Oracle Communications BRM is tested thoroughly before system
cut-over. One of the major benefits of using Oracle GoldenGate for near zero downtime upgrades is
the ability to thoroughly test the new Oracle Communications BRM release prior to go-live, since the
testing and upgrade process does not interrupt end users’ access to the old version . To accomplish
this, Oracle Database’s Flashback feature can be utilized to test the new Oracle Communications BRM
release. Using Oracle Flashback provides the following benefits.
 True Testing. Instead of having to test new features and functionality in a separate test
environment, testing can occur on real production data. It is a best practice to run billing,
invoicing and General Ledger reports against the source and target Oracle Communications
BRM systems and compare the results to ensure everything is working correctly.
 More Time to Test. Because of the amount of downtime required to perform the previous
approaches of in-place or parallel system run Oracle Communications BRM upgrades, only a
short window of time was spent testing the upgraded production environment before it
needed to go live. The Oracle GoldenGate near zero downtime upgrade approach allows for
more time to test, minimizing the risk to the business.
 Data Integrity Maintained. Oracle GoldenGate uses checkpoint logic to know exactly the
last transaction applied to the target database and where to restart. Combining this
functionality with Oracle Flashback, users are allowed to thoroughly test the new Oracle
Communications BRM release making changes to the target database. When testing has
completed, Oracle Flashback rolls those changes back, Oracle GoldenGate is restarted, and
the databases are re-synchronized again in real time.

The following is an example of how Oracle Flashback can be used for testing:
Before running billing on the source Oracle Communications BRM system, the GoldenGate Delivery
process would be stopped on the target Oracle Communications BRM system and an Oracle Flashback
checkpoint would be created. Billing would then be run against the source Oracle Communications
BRM system as well as the target Oracle Communications BRM system. The resulting billing
statements would be compared and validated between the source and target Oracle Communications
BRM systems to ensure the new target Oracle Communications BRM release is working correctly.
Once billing tests have been completed, the target Oracle Communications BRM database would be
rolled back to the Oracle Flashback checkpoint and the GoldenGate Delivery process would be
restarted to re-synchronize the source and target Oracle Communications BRM databases.

10
 Near Zero Downtime Upgrade for Oracle Communications Billing and Revenue Management Using Oracle GoldenGate

Roll-out Phase
The process of deploying the near zero downtime upgrade solution into production varies by
deployment model; the deployment model chosen will dictate the number of steps involved. The
unidirectional model is always completed first with the option to setup the reverse downgrade logic for
the failback model.

Unidirectional Model

At a high-level, the deployment process for a unidirectional model entails the following:
 Setup the target environment (hardware, storage, network, etc).
 Start the Oracle GoldenGate Capture process on the source Oracle Communications BRM
system.
 Copy the source database to the target database server and synchronize the databases in real-
time.
 Run the Oracle Communications BRM upgrade scripts on the target database.
 Configure the GoldenGate Delivery process with the upgrade logic and synchronize the
source and target databases.
 Execute test plans against the new target Oracle Communications BRM release to validate it’s
ready for cut-over.

If this were the final goal of the deployment, after testing, a cut-over would be scheduled to the new
Oracle Communications BRM target system.

Failback Model

Once the flow of changed data from the prior Oracle Communications BRM release to the new Oracle
Communications BRM release is established and tested, the failback data flow can be deployed. . The
failback option establishes the flow of changed data from the new Oracle Communications BRM
release to the prior, which includes downgrade transformations, before any users are active on the new
release of Oracle Communications BRM. Test cases can be injected into the new release, which will
then flow back to the prior release, providing the means to test the downgrade transformations.
For the failback deployment model, add the following steps to the steps outlined in the unidirectional
deployment model:
 Enable the Capture process on the new Oracle Communications BRM system.
 Enable the Delivery process with downgrade logic on the old Oracle Communications BRM
system.
 Perform downgrade logic testing.

Once the testing process is complete, the business can schedule a cut-over whereby users are moved to
the new Oracle Communications BRM release en masse. The failback option would be used only in the

11
 Near Zero Downtime Upgrade for Oracle Communications Billing and Revenue Management Using Oracle GoldenGate

worst-case scenario when the business needs to go back to the prior Oracle Communications BRM
release.

Ongoing Data Analysis

Data quality analysis is not a one-time event, and throughout the production roll-out, data must be
checked regularly to ensure correctness as it moves from one Oracle Communications BRM release to
another. Typically, this process coincides with Oracle Communications BRM regression testing, in
which users spot-check the data while testing Oracle Communications BRM.
In addition to counting the rows in the source and target databases and making visual comparisons, the
following two options can aid in the data analysis process. The first is a method to check referential
integrity within the Oracle Communications BRM data model, and the second is a data validation
method using Oracle GoldenGate Veridata, a high-speed data comparison tool.

Best Practice for Validating Data Consistency

As previously discussed, deploying Oracle GoldenGate for near zero downtime BRM upgrades
involves creating a live parallel copy of the source production BRM system, upgrading the target BRM
system to the new release, and keeping it in sync with the prior release using Oracle GoldenGate
before switchover. During the data movement, discrepancies can arise as result of infrastructure
problems, application errors, configuration errors, and unexpected user behavior. It is important to
identify and eliminate these inconsistencies before switchover to avoid operational, financial, or
regulatory risk.
Oracle GoldenGate Veridata is a high-speed data comparison solution that identifies and reports data
discrepancies between databases without interrupting ongoing business processes. Using this
application, users can quickly compare the contents of the source and target databases.
Oracle GoldenGate Veridata is able to compare the data contents of the key Oracle Oracle
Communications BRM base tables used in the replication process to determine if any data differences
exist. Differences between key tables would indicate that the transformation upgrade logic embedded
in Oracle GoldenGate was not correctly coded. If the source and target Oracle Communications BRM
systems are identical, this would indicate that the transformation results were coded correctly and
yielded the same values for the changed data as compared to the in-place upgrade scripts.

12
 Near Zero Downtime Upgrade for Oracle Communications Billing and Revenue Management Using Oracle GoldenGate

Conclusion
Today’s organizations cannot remain competitive with outdated solutions, yet they cannot afford to
experience extended downtime. Through robust data replication and migration capabilities, the Oracle
Communications BRM near zero downtime solution utilizing Oracle GoldenGate offers organizations
the ability to leverage the latest Oracle Communications BRM features and functionality from Oracle
without interrupting daily operations. The Oracle Communications BRM near zero downtime solution
provides customers with two approaches when upgrading their Oracle Communications BRM system
to the latest release: the unidirectional model or the more advanced failback model. Regardless of the
chosen approach, the Oracle Communications BRM near zero downtime solution eliminates downtime
and enables continuous access to mission-critical systems, thereby removing the most common barrier
to Oracle Communications BRM upgrades. Now Oracle Communications BRM customers can quickly
implement the latest application functionality to optimize their business processes and leverage a more-
efficient system that delivers higher business value.

13
Near Zero Downtime Upgrades for Oracle
Communications Billing and Revenue
Management Using Oracle GoldenGate
Januarary 2012
Copyright © 2012, Oracle and/or its affiliates. All rights reserved. This document is provided for information purposes only and the
contents hereof are subject to change without notice. This document is not warranted to be error-free, nor subject to any other
warranties or conditions, whether expressed orally or implied in law, including implied warranties and conditions of merchantability or
fitness for a particular purpose. We specifically disclaim any liability with respect to this document and no contractual obligations are
formed either directly or indirectly by this document. This document may not be reproduced or transmitted in any form or by any
means, electronic or mechanical, for any purpose, without our prior written permission.

Oracle Corporation
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners.
World Headquarters
500 Oracle Parkway
Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used under license and
Redwood Shores, CA 94065
are trademarks or registered trademarks of SPARC International, Inc. AMD, Opteron, the AMD logo, and the AMD Opteron logo are
U.S.A.
trademarks or registered trademarks of Advanced Micro Devices. UNIX is a registered trademark licensed through X/Open
Worldwide Inquiries: Company, Ltd. 0112
Phone: +1.650.506.7000
Fax: +1.650.506.7200

oracle.com