CDC Transformation and Delivery

Data at the speed of business

2008 IBM Corporation

What is CDC
Change Data Capture

Capture data events in source database and move only the changes to the target Many different ways of doing CDC
Timestamps Triggers

API
Log-based

What fuels the IBM CDC Roadmap?

The widest breadth of functionality:

Batch/pull and real-time push processing Guaranteed delivery/transactional integrity Multiple topologies (peer to peer, 1 to many, many to 1, uni-directional, bidirectional) Homogeneous & heterogeneous data synchronization
Broadest range of sources and targets

Log-based capture agents for DB2 (on all platforms), Oracle, SQL Server, Sybase, IMS, VSAM, IDMS, ADABAS
Native/parallel applies for all RDBMS and JMS Multiple data delivery protocols (TCP/IP, JMS)
Industry leading performance and scalability

End to end throughput and low latency Parallel Apply to target system Low impact on source database systems
3

What fuels the IBM CDC Roadmap .

3000+ customers using the existing CDC products for;

HA/DR (DB back-up, fault tolerance) Real-time reporting/off-load querying Application Co-existence (migrations, upgrades, modernization)

eCommerce (web apps, portals, data distribution)

Dynamic Data Warehousing, Master Data Management
700+ people in engineering focused on Information Integration including 170+ focused on CDC technologies

The most comprehensive suite of data integration products

BoB transform / cleanse / discovery, metadata management, scalable performance, services enabled for SOA architectures 5000+ customers using Information Server components
4

The IBM Solution: IBM Information Server

Delivering information you can trust
IBM Information Server
Unified Deployment

Discover, model, and govern information structure and content

Standardize, merge, and correct information

Combine and restructure information for new uses

Capture, virtualize and move information for inline delivery

Unified Metadata Management

InfoSphere CDC Solution

Provides real-time change data capture and delivery for
Dynamic warehousing and real-time reporting Synchronization and replication Event detection Minimal impact on production systems High scalability and end-to-end performance Guaranteed data integrity Proven Heterogeneous support
Without impacting performance of production systems Developers Architects

DataMirror
Delivers real time changed data to Information Server, applications and targets or message queues

Key Value Proposition

LATENCY
1. Near zero latency for pervasive integration projects. 2. ETL can also deliver low latency but at what impact to product systems and mission-critical applications.

Low Impact

Low Latency

IMPACT
1. Reduces risk to operational systems. 2. Non intrusive to applications and databases. 3. Use of native DB logs, documented overhead of 2-5%. 4. No use of disk based staging or triggers. 5. Management easily integrated into existing IT operations. 6. Help reduce/manage operational windows.

Continuous Consistent Data Delivery

CONSISTENT DATA DELIVERY

1. Data pushed from source, delivered in continuous stream, continuous with business operations. 2. Transaction consistency maintained to preserve units of work, referential integrity. 3. Full transaction granularity, before and after image of all transactional changes. 4. Data event aware, can be used to trigger specific business processes. 5. Fault tolerance, recover to last committed transaction.

Architecture
Publisher

Java-based GUI for admin & monitoring

Subscriber

Database

ODS TCP/IP Audit JMS

Journal Log Redo/Archive Logs

Source Engine And Metadata

Target Engine And Metadata

Business Process Flat files

Databases Oracle, DB2, DB2 UDB, SQL Server, Sybase, Teradata, Netezza, PointBase IMS, VSAM, IDMS, Adabas, DataCom - Classic

Direct to existing ETL

Platforms z/OS, System i5, Red Hat and SUSE Linux, AIX, HP/UX (PA-RISC and Itanium), Solaris SPARC, Tru64 UNIX, Windows Messaging Middleware MQSeries, Sun Open Message Queue (JMS), TIBCO, BEA AquaLogic, Oracle Fusion Middleware
8

Use Cases
Customer examples
9
2008 IBM Corporation

1. Building A Low Latency ODS for Operational Reporting and Auditing

Solution deployed to improve visibility into lines of business for organizations with Operational BI and Data Auditing requirements
Production Server
ERP

Native DB

Operational Data Store

OLTP

Log

Manufacturing

Production Server
Finance

ODS

Native DB

OLTP

Each OLTP insert, update and delete operation can be stored as an insert, update and delete to maintain synchronized copy of data. All OLTP insert, update and delete operations can be stored as inserts to maintain complete transaction history. Add relevant information such as timestamp, transaction type, source system id, and id of user who changed the transaction.

Log

Manufacturing

10

2. Complementing An Existing ETL Technology

Solution deployed to improve visibility into lines of business (i.e. Dynamic Warehousing) and help manage impact concerns caused by ETL on mission critical systems

Production Server
Point Of Sale

ETL Server

Data Warehouse

Native

OLTP

DB Log

Continuous

Stage
Retail

ETL

Scheduled Batch

EDW

Stage can be: 1. Relational Table 2. Flat File 3. Message Queue 4. Direct to ETL

Complementary ETL Technologies: 1. Informatica Power Center 2. Business Objects Data Integrator 3. Ab Initio 4. IBM DataStage (has native integration)

11

3. Continuous Feed Of A Business Intelligence Appliance

Solution deployed to improve visibility into lines of business by combining the cost/performance benefits of a BI Appliance with real-time data feeds.

Production Server
ERP

Staging Server

Appliance Nodes/Cluster

Native

OLTP

DB Log
Manufacturing

CDC Continuous (to Appliance)

CDC Stage
Flat File

Appliance Load API

Appliance

Flat file containing transaction changes viewed as an external file to the appliance. Load threshold based on # of Transactions or time interval. Once threshold reached, call appliance load API to bulk load transactions into appliance.

Supported Appliances 1. Teradata 2. Netezza 3. GreenPlum 4. Paraccel 5. IBM Balanced Warehouse

12

Data Event Synchronization via an Enterprise Service Bus

Solution deployed to provide real time data feeds for SOA and application Solution deployed to provide real time data feeds for SOA and application integration business requirements. integration business requirements.

Production Server
Billing

Production Server
CRM

E
Native

CDC Continuous

Queue 1

S
Queue 1

CDC Continuous

OLTP

DB Log

B
ETL
Telco Telco

OLTP

CDC/Replication Process Other Technology CDC/Replication License A license would reside on the server that hosts the message oriented middleware.
13

Complimentary ESB Technologies: 1. IBM MQ Series 2. TIBCO Business Works 3. BEA Aqualogic 4. WebMethods Fabric

5. e-Commerce Application Synchronization

Solution deployed to provide continuous customer, sales and inventory visibility in web base e-commerce applications.
Website Orders

Native DB

OLTP

Production Server
Inventory

Log

Corporate

Native

OLTP

DB Log

Point Of Sale
Retail
Native DB Log

OLTP

Provides continuous bi-directional synchronization between web based applications and mission critical business applications.

Downtown Store

Helps organizations improve customer online shopping experience with improved visibility into inventory and customer shopping activities. 14

6. Data Synchronization for Upgrades, Migrations and Workload Balancing

Solution deployed to help IT support application, database and platform migrations.

Production Server
ERP ERP

Testing Server

Native

Upgrades, Migrations
Workload Balancing

Native DB Log

OLTP

DB

OLTP

Log

Manufacturing

Manufacturing

Keep data synchronized between current production server and a server deployed to test a new application upgrade/version, or a hardware/OS upgrade. Workload balancing capability (i.e. master to master support) allows database instances to remain synchronized where dual or double data entry is a requirement (i.e. data entry occurring on both systems at the same time).

15

7. Offloading Production Query & Reporting Cycles

Solution deployed to allow organizations to offload the impact of query and reporting to a non mission critical system.
Production Server(s)
Finance 1

OLTP

Native DB Log

Services

Reporting Server
Finance 2

Table Copy

OLTP
Services

Report Query

Finance 3

Native

Reporting server can also be used for consolidation requirements i.e. consolidating financials from multiple branches into a single corporate instance. Replication frequency generally varies from continuous (near real-time) to periodic. Table level refresh or copy can be used in addition to log based change data capture.

OLTP

DB Log

Services

16

8. Data Backup And Availability

Solution deployed to allow organizations to backup copies of critical data for recovery where a full disaster solution is not a requirement.
Production Server
Finance
CDC Continuous (to backup instance)

Availability Server

Native

OLTP

DB
Log

Backup

Partition 1

Backup
Partition 2

Availability of data only, does not support DDL replication. Exact image replication to produce a backup copy on a separate server or in a different partition on the same server. A separate license is not required for each partition used on the production server.

17

Thank You

18

2008 IBM Corporation