8398 Goldengate Integrated Capture 1888658

Copyright 2012, Oracle and/or its affiliates. All rights reserved.
1
Copyright 2012, Oracle and/or its affiliates. All rights reserved. 2
Extracting Data in Oracle
GoldenGate Integrated
Capture Mode
J oydip Kundu
Director, Software Development
Enterprise Replication
Oracle Corp
The following is intended to outline our general product
direction. It is intended for information purposes only,
and may not be incorporated into any contract.
It is not a commitment to deliver any material, code, or
functionality, and should not be relied upon in making
purchasing decisions. The development, release, and
timing of any features or functionality described for
Oracles products remains at the sole discretion of
Oracle.

Program Agenda
Introduction to Oracle GoldenGate
Overview of Integrated Capture
Benefits of Integration
Deployment Options
Operational Aspects of Integrated Capture
Summary
Introduction to Oracle
GoldenGate
Log-based
Changed Data
Data Integrator
New DB/
HW/OS/APP
Fully Active
Distributed
DB
Reporting
Database
Data
Warehouse
ODS
Zero Downtime Upgrade
& Migration
Query & Report Offloading
Data Synchronization within
the Enterprise
Real-time BI, Operational
Reporting, MDM
Event Driven Architecture,
SOA
High Availability/ Disaster
Recovery
Message
Bus
Oracle & Non-Oracle
Database(s)
Message Bus
Legacy Systems
Oracle GoldenGate
Low-Impact, Real-Time Data Integration & Transactional Replication
Global
Data
Centers
Oracle GoldenGate
Differentiators
Flexible Performance Reliable
Low latency for data
movement

Low-impact capture,
routing, transformation,
and delivery of
transactional data

Modern, open, modular
architecture

Supports heterogeneous
sources and targets

Agile and adaptable to
modern data integration
architectures

Maintains transactional
integrity

Resilient against
interruptions and failures

Fast Recovery after
process interruptions.

How Oracle GoldenGate Works
Source
Oracle & Non-Oracle
Database(s)
Target
Oracle & Non-Oracle
Database(s)
Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction
logs. New with V.11.2.1, GoldenGate offers two options for capture for Oracle; Classic & Integrated Capture
Trail: stages and queues data for routing.
Pump: distributes data for routing to target(s).
Route: data is compressed, encrypted for routing to target(s).
Delivery: applies data with transaction integrity,
transforming the data as required.
Capture
Trail
Files
Pump Delivery
LAN / WAN / Internet
Over TCP/IP
Trail
Files
How Oracle GoldenGate Works
Source
Oracle & Non-Oracle
Database(s)
Target
Oracle & Non-Oracle
Database(s)
Capture: committed transactions are captured (and can be filtered) as they occur by reading the transaction
logs. New with V.11.2.1, GoldenGate offers two options for capture for Oracle; Classic & Integrated Capture
Trail: stages and queues data for routing.
Pump: distributes data for routing to target(s).
Route: data is compressed, encrypted for routing to target(s).
Delivery: applies data with transaction integrity,
transforming the data as required.
Capture
Delivery
Trail
Files
Pump
Trail
Files
Pump
Delivery
Capture
Bi-directional
LAN / WAN / Internet
Over TCP/IP
Trail
Files
Trail
Files
Program Agenda
Deployment Options
Summary
Integrated Capture:
What is it?

Classic Capture
Redo Parsing
Oracle GoldenGate: Classic Capture
Memory Manager
SQL Interface
Filtering
Formatting
Transformation and
Mapping
Checkpoint Manager
Transformation
Engine
Trail File
Generation
Messaging Subsystem
Database Access
Parser
Metadata
Manager
GoldenGate
Event Log
GoldenGate
Checkpoint File
GoldenGate
Trail Files
GoldenGate
Parameter File
Database
Archive/Redo Logs
Oracle GoldenGate: Integrated Capture

Integrated Capture
Memory Manager
SQL Interface
Filtering
Formatting
Transformation and
Mapping
Checkpoint Manager
Transformation
Engine
Trail File
Generation
Messaging Subsystem
Database Access
Parser
Metadata
Manager
LCR
Processor
Database
LogMining
Server
GoldenGate
Event Log
GoldenGate
Checkpoint File
GoldenGate
Trail Files
GoldenGate
Parameter File
Program Agenda
Deployment Options
Summary
Integrated Capture:
Enhancing Redo-Based
Capture Support
Oracle GoldenGate (Benefits of Integration)
OGG-11.2-CC OGG-11.2-IC

Data
Type
Compression No Full Support
(Basic-, OLTP- and EHCC)
PDML/XA on RAC
Distributed Txn
No

Yes
Basicfile LOBs DMLs from redo
DBMS_LOB via fetch
Yes
Securefile LOBs

Fetch Based Full Support
Fetch: DEDUP/NOLOGGING
XML XML-CLOB : Redo
Binary XML: Fetch
XML-OR: No
XMLType Table: No
Full Support
Oracle GoldenGate (Benefits of Integration)
OGG-11.2-CC OGG-11.2-IC
Integration
with
Database
features
ASM Yes (Setup required:
DBLOGREADER)
Transparent Support
RMAN Yes (Setup: Register
Extract)
Transparent Support
Data Guard Limited Transparent Support
RAC User intervention required
for addition/removal of
instances
Transparent Support
Program Agenda
Deployment Options
Summary
Supports multiple deployment configuration
On-Source : Source database and Integrated
Capture process are in the same machine
Downstream : Integrated Capture runs on
different database typically on different
machines

Easy transitions for existing GoldenGate
customers
Customers may choose which option they prefer
based on their requirements.

Integrated Capture: Deployment Overview
On Source
Downstream

Architecture Real time mode
Source database and Integrated Capture process are in the same machine

User/Application changes
Record
changes
Capture
changes
Redo Logs
Capture
Integrated
Extract

Oracle Database
Trail
File
..
Source Machine
LCR1
LCR2
LCR3
.
.
.
Logical Change
Records
On-Source Deployment of Integrated Capture
Register with database
GGSCI>DBLOGIN USER dbuser PASSWORD dbpasswd
GGSCI>REGISTER EXTRACT ext1 DATABASE
Must do this before you can add Extract
Can take a few minutes on large database
Add Extract
GGSCI>ADD EXTRACT ext1 INTEGRATED TRANLOG, BEGIN NOW
All other steps remain the same
No change required parameter file
Supported Features Database Version
Oracle Database Versions Supported in OGG 11gR2

Cap t u r e
( a k a C l a s s i c
C a p t u r e )
10.2 Supported
11.1.0.7 TDE/TSE, Fetch support for SecureFiles

I n t eg r at ed
Cap t u r e
10.2 * Supported
11.1.0.7 * Supports TDE/TSE
11.2.0.2 * Supports TDE/TSE,XA RAC, Compression
Fetch support for Securefiles, ADTs, VARRAYS, Nested
Tables, Object Tables
11.2.0.3 Supports TDE/TSE, XA RAC, Compression, XML OR, XML
Binary
Fetch support for Securefiles, ADTs, VARRAYS, Nested
Tables, Object Tables

Architecture Real Time Downstream mode
Record
changes
Redo Logs
Ca
ptu
re
Integrated
Extract

Oracle Database
Trail
File
Source Machine
Source database and Integrated Capture process are in different machine

Standby
Redo Logs

Oracle Database
Downstream Machine
..
Continuous REDO
Shipping
Capture
changes
LCR1
LCR2
LCR3
.
.
.
Logical Change
Records
Capture

Architecture Downstream Archive Log mode
Record
changes
Redo Logs
Ca
ptu
re
Integrated
Extract

Oracle Database
Trail
File
Source Machine
Source database and Integrated Capture process are in different machine

Archive Logs

Oracle Database
Downstream Machine
..
Capture
changes
LCR1
LCR2
LCR3
.
.
.
Logical Change
Records
Capture
Integrated Capture: Downstream Deployment
Database
Release
Redo
Compatibility
On-Source
Deployment
Downstream
Deployment
10.2.0.5+ 10.2+ No No support for
XML/Objects/Compression/
TDE/TSE
11.1.0.7+ 11.0+ No No support for
XML/Objects/Securefiles/
Compression
11.2.0.1/2 11.2+ No No support for XML/Objects
11.2.0.3 11.2+ All features All features
New Concepts (Downstream Deployment)
GGSCI
Source Database (Existing)
GGSCI>DBLOGIN userid PASSWORD pwd
Mining Database (New)
MININGDBLOGIN USERID mdbuser, PASSWORD mpwd
Defaults to DBLOGIN if not specified
Used for creating logmining server
Mandatory for downstream deployment
Must have privileges granted through
SQL >EXEC DBMS_GOLDENGATE_AUTH.GRANT_ADMIN_PRIVILEGE( -
grantee =>'gg_admin', -
privilege_type =>capture, grant_select_privileges=>true,
do_grants =>TRUE);

New Concepts (Downstream Deployment)
Extract
Source Database (Existing)
USERID user PASSWORD password
Mining Database (New)
TRANLOGOPTIONS MININGUSER USERID mdbuser, MININGPASSWORD mpassword
Used for interacting with the logmining server
Mandatory for downstream deployment
Must have privileges granted through
SQL >EXEC DBMS_goldengate_AUTH.GRANT_ADMIN_PRIVILEGE( -
grantee =>'gg_admin', -
privilege_type =>capture, grant_select_privileges=>true,
do_grants =>TRUE);

Downstream Deployment of Integrated Capture
Set up Mining User Account
Use dbms_goldengate_auth.grant_privileges
GRANT SELECT ON V_$DATABASE to <user>
Configure archival of local redo
Make sure downstream mining database is in archivelog mode
SQL>ALTER SYSTEM SET LOG_ARCHIVE_DEST_1='
LOCATION=/home/arc_dest/local
VALID_FOR=(ONLINE_LOGFILE,PRIMARY_ROLE) ' ;
SQL>ALTER SYSTEM SET LOG_ARCHIVE_DEST_1 ENABLE;

Downstream Deployment of Integrated Capture:
Archived Log Only Mode
Prepare Source Database to Send Redo
Set DG_CONFIG
SQL>ALTER SYSTEM SET LOG_ARCHIVE_CONFIG
=' DG_CONFIG=(dbms1, dbmscap) '
Set up redo transport
SQL>ALTER SYSTEM SET LOG_ARCHIVE_DEST_2 =
'SERVICE=DMBSCAP.EXAMPLE.COM ASYNC NOREGISTER
VALID_FOR=(ONLINE_LOGFILES, PRIMARY_ROLE)
TEMPLATE=/usr/oracle/log_for_dbms1/dbms1_arch_%t_%s_%r.log
DB_UNIQUE_NAME =dbmscap
SQL>ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2 =ENABLE

Archived Log Only Mode
Set up Integrated Capture on DBMSCAP
Register Extract with Downstream Database
GGSCI>DBLOGIN userid ggasadm1 password ggs123
GGSCI>MININGDBLOGIN userid ggadmcap password ggs123
Create Extract at the downstream mining database
GGSCI>ADD EXTRACT ext1 INTEGRATED TRANLOG BEGIN NOW
Edit Extract Parameter file (ext1.prm)
USER ggadm1@dbms1 password
MININGUSER ggsadmcap@dbmscap MININGPASSWORD
Real Time Mode
Prepare Mining Database to Receive Redo
Create Standby Redo Logs (SRL)
SQL>ALTER DATABASE ADD STANDBY LOGFILE GROUP 3
Must have 1 more SRL per redo thread
Use V$STANDBY_LOG to verify
Set archival for SRLs
'LOCATION=/home/arc_dest/srl_dbms1
VALID_FOR=(STANDBY_LOGFILE, PRIMARY_ROLE) ';

SQL>ALTER SYSTEM SET LOG_ARCHIVE_DEST_STATE_2 =ENABLE
Real Time Mode
Prepare Source Database to Send Redo
Set DG_CONFIG
SQL>ALTER SYSTEM SET LOG_ARCHIVE_CONFIG
=' DG_CONFIG=(dbms1, dbmscap) '
Set up redo transport
'SERVICE =DBMSCAP.EXAMPLE.COM ASYNC NOREGISTER
VALID_FOR=(ONLINE_LOGFILES, PRIMARY_ROLE)
DB_UNIQUE_NAME=dbmscap ';
Real Time Mode
Set up Integrated Capture on DBMSCAP
Register Extract with Downstream Database
GGSCI>DBLOGIN userid ggasadm1 password ggs123
GGSCI>MININGDBLOGIN userid ggadmcap password ggs123
Create Extract at the downstream mining database
GGSCI>ADD EXTRACT ext1 INTEGRATED TRANLOG BEGIN NOW
Edit Extract Parameter file (ext1.prm)
USER ggadm1@dbms1 password
MININGUSER ggsadmcap@dbmscap MININGPASSWORD ggs123
TRANLOGOPTIONS INTEGRATEDPARAMS (downstream_real_time_mine Y)
Redo Log Setup
Things to Remember
Foreign Archived Logs should not be archived in the FRA (Fast Recovery Area)
Real time mining can be active for only one source database
Each SRL must be as large as the largest redo log file at the source database
(Recommendation: Use redo logs of uniform size)
If you have n redo threads each with m redo log groups, you will need n*(m+1)
redo log groups at the downstream database
DBIDs of the source databases must be unique
New Parameters for Integrated Capture
TRANLOGOPTIONS FETCHPARTIALLOB:
Use it when replicating to non-Oracle databases
Replicat is older than 11.2.1.0.0
Fetches full LOB for partial LOB operations
TRANLOGOPTIONS FETCHPARTIALXML
Use it when replicating to non-Oracle databases
Replicat is older than 11.2.1.0.0
Fetches full XML instead of Oracle proprietary XML difference

Program Agenda
Deployment Options
Summary
TRANLOGOPTIONS INTEGRATEDPARAMS:
Example: INTEGRATEDPARAMS (max_sga_size 1024, parallelism 2)
max_sga_size
Amount of shared memory (in MB) to use in the logmining server
Comes from streams_pool
Add up max_sga_size from all Extracts running on a given instance
Make sure you leave 25% of streams_pool for other database
activities like AQ, Datapump
We default it to 1024 MB

TRANLOGOPTIONS INTEGRATEDPARAMS:
parallelism
Controls the number of preparer processes
Defaults to 2
TRANLOGOPTIONS CHECKPOINTRETENTIONTIME
Defaults to 3 days
how long checkpoints taken by the logmining server are saved

Unneeded Parameters for Integrated Capture
THREADS (RAC Integration)
ASMUSER, ASMBUFSIZE, DBLOGREADER (ASM Integration)
DECRYPTPASSWORD (TDE/TSE Integration)

Integrated Capture:
Operational Things to Remember
Cannot alter SCN or Timestamp of Extract process before REGISTER
Logmining Server checkpoints are removed periodically
RDBMS alert log is a rich source of information
Important MOS Notes
Specific patches for Integrated Capture: 1411356.1
Integrated Capture health check script 1448324.1
OGG Best Practices: Configuring Downstream Integrated Capture 1485620.1
Performance Tuning for OGG 1488668.1

Looking at the alert log

GoldenGate CAPTURE CP01 for OGG$CAP_EXTWELL started with pid=112, OS id=27802
Starting persistent Logminer Session with sid =6 for GoldenGate Capture
OGG$CAP_EXTWELL
LOGMINER: Parameters summary for session#=6 <= This remains the same
LOGMINER: StartScn: 2883760144139 (0x029f.6d7fbf0b)
LOGMINER: SkipScn: 2883751645398 (0x029f.6cfe10d6)

Looking at the alert log (contd)
LOGMINER: session#=6 (OGG$CAP_EXTWELL), reader MS03 pid=149 OS id=27806 sid=3417 started
LOGMINER: session#=6 (OGG$CAP_EXTWELL), builder MS06 pid=158 OS id=27808 sid=6798 started
LOGMINER: session#=6 (OGG$CAP_EXTWELL), preparer MS04 pid=181 OS id=27810 sid=456 started
LOGMINER: session#=6 (OGG$CAP_EXTWELL), preparer MS05 pid=229 OS id=27812 sid=3431 started

LOGMINER: Begin mining logfile during dictionary load for session 6 thread 1 sequence 250, +FRA/
LOGMINER: End mining logfile during dictionary load for session 6 thread 1 sequence 250, +FRA/

LOGMINER: Begin mining logfile for session 6 thread 1 sequence 259, +FRA/
LOGMINER: End mining logfile for session 6 thread 1 sequence 259, +FRA/

RMAN & Recovery Area Integration
If mining on local database
Archived logs can be staged in Fast Recovery Area
RMAN will not delete any archived logs that Integrated Capture
needs unless space is exhausted
Note we have to delete otherwise database availability will be
compromised
(alert log): WARNING: The following archived logs needed by
Streams/GoldenGate capture process are being deleted to free space in the
recovery area....
If mining on downstream database
Cannot use Fast Recovery Area to stage foreign archived logs

Program Agenda
Deployment Options
Summary
Additional data type support via redo
Compression (Basic, OLTP- and EHCC)
PDML/XA on RAC and Distributed Transaction
Basicfile and Securefile LOBs (including compression and encryption)
XML (CLOB, Binary and OR)
Integrated with Database features
RAC
ASM
Data Guard
TDE/TSE

Integrated Capture: Summary
Join the Data Integration Community
Twitter
twitter.com/ORCLGoldenGate
Facebook
facebook.com/OracleDataIntegration
Oracles Data GoldenGate YouTube channel
youtube.com/oraclegoldengate
Oracle.com/goto/dataintegration
LinkedIn
Oracle Data Integration
Oracles Data Integration blog
blogs.oracle.com/dataintegration
Key Oracle GoldenGate Sessions at OOW
Monday October 1
st

Real World Zero Downtime Operations Customer Panel
Oracle GoldenGate 11gR2 New Features
1:45pm Moscone West 3005
Tuesday October 2
nd

Oracle GoldenGate and Oracle Data Guard: Working Together Seamlessly
Real World Operational Reporting Customer Panel
High-Volume OLTP with Oracle GoldenGate: Best Practices from Comcast
Everything You Need to Know about Monitoring Oracle GoldenGate
10:15am Moscone South305
11:45am Moscone West 3005
1:15pm Moscone West3022
5pm Moscone West 3005
Wednesday October 3
rd

Best Practices for High Availability with GoldenGate on Oracle Exadata
Oracle GoldenGate Case Study: Real-Time Operational Reporting Deployment
Oracle GoldenGate Best Practices for Conflict Detection and Resolution
Tuning and Troubleshooting Oracle GoldenGate on Oracle Database
11:45am Moscone South 102
5pm Moscone South 102
Oracle GoldenGate Hands-On-Labs
Wednesday October 3rd
Introduction to Oracle GoldenGate Veridata 10:15am Marriott Marquis - Salon 1/2
Thursday October 4th
Deep Dive into Oracle GoldenGate 11:15am Marriott Marquis - Salon 1/2
After OpenWorld, visit oracle.com/goto/availability
Key HA Sessions and Demos by Oracle Development
Monday, 1 October Moscone South
12:30p Oracle Data Guard Zero-Data-Loss Protection at Any Distance, 300
12:30p Future of Exadata: OLTP, Warehousing, and Consolidation, 104
1:45p Automating ILM with the Latest Database Technology, 300
1:45p Extracting Data in Oracle GoldenGate Integrated Capture Mode, 102
3:15p Maximize Availability with the Latest Database Technology, 303
3:15p Maximize Enterprise Availability with the Latest DB Technology, 303
4:45p Mission-Critical Oracle Exadata OLTP Deployment at PayPal, 300
4:45p Temporal Database Capabilities with the Latest DB Technology, 300
Tuesday, 2 October Moscone South
10:15a Database Tables to Storage Bits: Data Protection Best Practices, 300
10:15a GoldenGate & Data Guard: Working Together Seamlessly, 305
11:45a Active Data Guard Zero-Downtime Database Maintenance, 300
11:45a Using Automatic Storage Mgmt with the Latest DB Technology, 301
1:15p The Four Ts of RMAN: Tips, Tuning, Troubleshooting, and ?, 102
5:00p Maximum Availability Architecture Best Practices for Exadata, 303

Wednesday, 3 October Moscone South
10:15a Operational Best Practices for Oracle Exadata, 102
10:15a Maximize Availability by Minimizing Disruption for End Users
and Application, 301
11:45a Whats New in the Latest Generation of Oracle RAC, 301
11:45a Best Practices for HA w/ GoldenGate on Oracle Exadata, 102
1:15p Oracle Secure Backup: Integration Best Practices with
Engineered Systems, 300
1:15p Application MAA Best Practices on Oracle Private Clouds, 200
5:00p Tuning &Troubleshooting Oracle GoldenGate on Oracle, 102
Thursday, 4 October Moscone South
11:15a Integrate Your Globally Distributed IT for Key
Cloud Computing Benefits, 300
12:45p Backup and Recovery of Oracle Exadata: Experiences
and Best Practices, 300

Demos Mon 10:00a-6:00p - Tue 9:45a-6:00p - Wed 9:45a-4:00p
Oracle Maximum Availability Architecture, S-011
Oracle GoldenGate 11gR2 New Features, S-239
Oracle Database 12c: Global Data Services, S-010
Oracle Database 12c Application Continuity - S-009

Oracle Secure Backup, S-014
Oracle Active Data Guard, S-007
Oracle Recovery Manager and Oracle Flashback Technologies, S-019
Oracle Real Application Clusters and Oracle RAC One Node - S-008
Oracle Database 12c Xstream, Streams, Advanced Queueing, S-018

Brings GoldenGate extract closer to the RDBMS engine
Leverages Oracle Database Logmining API
Seamless traversal of different branches of redo
Parallelism in mining and transforming redo records
Transparent mining of different threads in a RAC
Future enhancement in the RDBMS area will be supported quicker in the product
Multi Threaded
Pipelined architecture.
Performance gain from parallel processing

Integrated Capture: Overview
Integrated Capture: Deployment Options
On-Source or Local Deployment
Capture runs on the same database
Real time capture of changes
Source database must be at 11.2.0.3+
Downstream or Remote Deployment
Use redo transport to ship redo logs
No overhead on source
Can ship redo from multiple source databases
Only one source can be mined in real time
Source can be in 10.2.0.5+
Downstream database must be at 11.2.0.3+ , same platform as source

Description Category
Integrated Capture - What You Need To Know
Exadata Capture support for EHCC compression
Compression Capture support for OLTP, Segment compression
Distributed
Transactions
XA-RAC, PDML
RAC Dynamic Node Awareness
New Datatypes XML OR, XML Binary
LOB Full and Partial read from REDO log
Deployment Online and Downstream capture support
DDL
Captures tables created with column level password
specification

8398 Goldengate Integrated Capture 1888658

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

8398 Goldengate Integrated Capture 1888658

Hochgeladen von

Copyright:

Verfügbare Formate

Copyright 2012, Oracle and/or its affiliates. All rights reserved.

Das könnte Ihnen auch gefallen