ODIExperts.

com
The blog for Oracle Data Integrator ( ODI )

Home
Author
Blog
POST_LIST
POST_BY_ALPHABETICAL
POST_BY_DATE
POST_BY_CATEGORY
DRIVERS
ODI_JDBC_EXCEL
ODI_JDBC_ACCESS
ODI_HTML_EMAIL
SOURCE_CODES
JOBS
EMPLOYER_FORM
Contact Us

CDC CONSISTENT
Hi Friends,
Its a post about one of the most questioned subjects that we receive.
I hope you all enjoy once its a very detailed one.
CDC consistent is a concept where Journalizing is carried at the Model level . As you all are aware of the point that in ODI CDC are
of two types namely Simple and Consistent. We have already covered the Simple part early . You can visit this link for simple CDC
steps. http://odiexperts.com/changed-data-capture-cdc
CDC Consistent is great when we have tables in a schema related to each other via PK FK relationship, as CDC Consistent are
applied on the Model rather than on the Data store level as practiced in Simple.
Please find the step by step instruction of initial set up of CDC Consistent
Step 1. Edit the Model and select the appropriate Consistent Journalizing mode and select the appropriate Consistent KM.

Step 2. Add the data stores into CDC.

As you can see the all the data stores will have the orange clock shaped icon.
Step 3. Add the subscriber(s)

Step 4 Start the Journal

Once the Journalizing is successful the Orange clock icon will change to Green icon.

Testing the CDC Consistent on various scenarios.

Simple Insert is being carried at the Source Table.

Data being shown at the Journalizing data option.

Lets see if the Journalized data gets flow through.

The records does not seems to come through . Lets see the issue or the real reason behind it.

The source table for the journalizing is reading from the JV$ view, where as the data seems to be reading from the JV$D view.
The real reason behind why ODI reads from the JV$ rather than JV$D is that JV$ captures all the records from the JV$D by locking
the subscriber.
Consider this example say at 10 pm there are 3 Inserts , 5 Updates and 1 deletes are happening at the source side and thus gets

captured in the respective JV$D views .

We apply the Extend windows and Lock Journalize table and all the records in the JV$D gets captured into JV$.Now at that time
there is again a new 1 insert and so gets captured in the JV$D views. Since there is lock in the subscriber table . New data will not
be captured into JV$ .
Lets repeat the necessary steps and rerun the interface.
Step 1. Extend Window

Step 2. Lock Subscriber

Select the right subscriber

Step 3 Run the interface

Step 4 Unlock Subscriber

Step 5 Purge Journal

Once the data is been captured the journal is purged.

Lets see how we can process the above steps via Package.

Step 1 OdiWaitforLogData

Apart from the default setting provide the Logical Schema and either provide the table name or the CDC Set.
Step 2 Extend and Lock Subscribers the model
Drag and drop the Model into the Package and in the Type select Journalizing Model
and check mark Extend windows and Lock Subscribers

Step 3 Interfaces
Now call the interfaces.
Step 4 Unlock Subscribers and Purge Journal.
Next steps after the interface drag and drop the model and select Purge Journal and Unlock Subscribers.

Lets test the package . I am inserting another record and let see if the records come through .

Yes data has been inserted into the Target table.

Single Journalized table per interface Limit.

If there are n number of Journalized table in an interface , only one interface can be marked as Journalized data , even if they are
consistent based .

In order to avoid this limit , I have revised the code and re written it like this, so that we can use two journalized table without
requiring to read through the complete table of the other source.
Note There is a big assumption I am making that both the tables namely parent and child are dependent on each other via PK-FK
and so if there is an insert in one the of the Parent table then there will be insert into the child table too
<% if (odiRef.getDataSet(i, "HAS_JRN").equals("1")) { %>

<%=odiRef.getJrnInfo("FULL_TABLE_NAME")%>.JRN_FLAG IND_UPDATE <%} else {%>'I' IND_UPDATE <%}%>

from
<%=odiRef.getSrcTablesList("", "[WORK_SCHEMA].<? if (u0022[IND_JRN]u0022.equals(u00221u0022))
{?>JV$[TABLE_NAME] [TABLE_NAME] <?} else {?>[TABLE_NAME] [TABLE_NAME]<?}?> ", " , ", "")%>
where (1=1)
<%=odiRef.getJoin(i)%>
<%=odiRef.getFilter(i)%>
<%=odiRef.getSrcTablesList(" ", "AND [TABLE_NAME].<? if (u0022[IND_JRN]u0022.equals(u00221u0022))
{?>JRN_SUBSCRIBER = 'SUNOPSIS' /* AND JRN_DATE < sysdate */ <?} else {?><?}?> ", " ", "")%>
<%=odiRef.getGrpBy(i)%>
<%=odiRef.getHaving(i)%>
<%}%>

In order to use the above code in any KM.

Mark the parent Datastore with Journalized data store option.
Replace the code JRN_FLAG IND_UPDATE with <%=odiRef.getJrnInfo("FULL_TABLE_NAME")%>.JRN_FLAG IND_UPDATE
so it can use the Parent JRN_FLAG to accordingly drive the target.

Replace the code <%=odiRef.getFrom(i)%> with <%=odiRef.getSrcTablesList("", "[WORK_SCHEMA].<? if

(u0022[IND_JRN]u0022.equals(u00221u0022)){?>JV$[TABLE_NAME] [TABLE_NAME] <?} else {?>[TABLE_NAME]
[TABLE_NAME]<?}?> ", " , ", "")%>

so if the datastore is journalized then its associated JV$ view is used else the table name
Similarly replace <%=odiRef.getJrnFilter(i)%> with <%=odiRef.getSrcTablesList(" ", "AND [TABLE_NAME].<? if
(u0022[IND_JRN]u0022.equals(u00221u0022)){?>JRN_SUBSCRIBER = 'SUNOPSIS' /* AND JRN_DATE < sysdate */ <?} else
{?><?}?> ", " ", "")%>

Here for this example i have used the hard coded subscriber name of SUNOPSIS , you can use the getoption and pass the subscriber
name in the Interface option.
The main reason for re writing it this way is that , when we have only one table as journalized , ODI have to read through the other
child tables even though they are Journalized.
Lets see an example of the above code
SOURCE

Two new region id is inserts into Regions table with region id 7 and 8

Record is insert with country _id ID and region id 7

Looking at the combination and the Location parent driving then both will be inserted into JV$D views for the respective table.

TARGET
As you can see for the country id and region id 7 , the target table has correctly fetched the associated Region name .

Hope this article gives you a better understanding on the CDC Consistent type.
Thank you and please, keep visiting us.
Like

Tweet

Posted Date: March 27, 2011

Author: kdevendr
Comments: 17 Comments
Categories: Administration , Architecture , How to , Knowledge Modules , ODI , Technology , Tips and Tricks
Tags: CDC , CDC Consistent
Permalink

17 Comments
Leave a reply

1.
Do Trung Hieu
August 13, 2014 at 8:52 AM
Can I create index on J$ tables?
Reply

Cezar Santos
August 14, 2014 at 6:14 PM
Yes, and I suggest you to add the create step at JKM
Reply

2.
Jonas
May 29, 2014 at 4:01 AM
Hi, this is an old post, but hopefully, someone can still answer my question.
Ive tried following the instructions to modify the knowledge module to overcome the Single Journalized table per interface
Limit.
However, I get the following errors because of this line:
<%=odiRef.getSrcTablesList("", "[WORK_SCHEMA].JV$[TABLE_NAME] [TABLE_NAME] [TABLE_NAME]
[TABLE_NAME] , , , )%>
The error I am getting is:
com.sunopsis.tools.core.exception.SnpsSimpleMessageException: ODI-17517: Error during task interpretation.
Task: 17
java.lang.Exception: BeanShell script error: Sourced file: inline evaluation of: out.print(insert /*+ APPEND */ into ) ;
out.print(snpRef.getObjectName(L, . . . : Undefined argument: u00221u0022 : at Line: 8 : in file: inline evaluation of:
out.print(insert /*+ APPEND */ into ) ; out.print(snpRef.getObjectName(L, . . . : ( u00221u0022 )

It is complaining about the argument u00221u0022. Could someone show some light as to what this argument is and if this
is the correct value or a typo on the example show?
Much appreciated
Reply

Jonas
May 29, 2014 at 4:04 AM
Sorry, the code that I am getting problem with is:
<%=odiRef.getSrcTablesList(" ", "AND [TABLE_NAME].JRN_SUBSCRIBER = SUNOPSIS /* AND JRN_DATE <
sysdate */ , , )%>
Also, we are using ODI 11g.
Reply

Jonas
May 29, 2014 at 4:06 AM
<%=odiRef.getSrcTablesList("", "[WORK_SCHEMA].JV$[TABLE_NAME] [TABLE_NAME]
[TABLE_NAME] [TABLE_NAME] , , , )%>
Reply

3.
Mario Ciappara
January 30, 2014 at 8:48 AM
I have followed the above post to implement Consistent CDC using the latest ODI 12c and JKM Oracle 11g Consistent
(Streams) KM. I would like to draw the attention to those users trying to implement the same kind of CDC, that when one
chooses the KM in question for a Data Model, one needs to change the Validate Option from default: true to false.
The reason behind this change is to ignore validation errors coming from the KMs code when trying to re-created the tables,
PKs etc. during a subsequent Start Journal task. The first Start Journal task on any datastore within the data model creates
the necessary tables with the STREAMS_OBJECT_GROUP prefix. Subsequent Start Journal tasks on other datastores will
try to re-create the same tables and if the Validate option is set to true it will raise errors causing the start Journal task to
fail.
Reply

Cezar Santos
January 31, 2014 at 11:56 AM
Thank you Mario!!!!
Reply

4.
Rao
November 22, 2013 at 4:50 AM
Hi,
Please reply on the above request if anyone have an idea.

Thanks in advance.
Regards,
Archana
Reply

5.
Rao
November 20, 2013 at 11:51 AM
Hi,
There is one table which having foreign key constraints on another schema table.Then can i use jk,m consistent.if yes,how?
Please advice on priority.
Thanks,
Rao
Reply

6.
hari
August 5, 2013 at 5:04 AM
can u please tell me where to replace above code in KM?
cheers,
Hari.
Reply

kdevendr
August 5, 2013 at 1:25 PM
you will need to use this code to load into i$ table.
Reply

7.
Joe
September 26, 2011 at 12:37 AM
I have got this situation, whereby I need to have consistent CDC. The form of consistent required is that for 2 tables of PK-FK
relationship, either both will be journalized or non.
For example, if theres changes to a set of parent-child records, it will be journalized. If there changes to the child record, but
no changes to the respective parent record, it will not be journalized.
Ive tried your steps on this webpage multiple times but it doesnt seems to work the way I want. Is there any details which I
need to specifically watch out for?
Thank you
Reply

8.
Spidy
July 6, 2011 at 2:02 PM

Cezar, devendra,
can we use ODI for loading the data from oracle databse to coherence cache.
If yes, then details pls.
Reply

9.
asmaa
June 8, 2011 at 2:22 AM
Hi Cezar,
in your post (SINGLE Journalized table per interface at ODI 11) you said(
Replace the code JRN_FLAG IND_UPDATE with .JRN_FLAG IND_UPDATE
Replace the code with <%=odiRef.getSrcTablesList("", "[WORK_SCHEMA].JV$[TABLE_NAME] [TABLE_NAME]
[TABLE_NAME] [TABLE_NAME] , , , )%>)
I searched in all kms that in ODI 10.1.3 I didnt see (odiref.getFrom(i))&(odiRef.getJrnfilter(i))
where can I find (odiref.getFrom(i))&(odiRef.getJrnfilter(i)) in odi10.1.3 KM ?
THANK YOU
Reply

10.
asmaa
June 7, 2011 at 5:14 AM
Hi cezar
I think odi 11g use (odiref.getFrom(i))&(odiRef.getJrnfilter(i))in its kms
but I want this issue in ODI10.1.3. which use (snpRef.) in its kms
please I want it in detail
its very important for my project
THANK YOU VERY MUCH
Reply

11.
Leo
April 15, 2011 at 7:52 AM
Is this for 10g or 11g? Im trying to programmaticly drop journal(remove J$ and jv$). Any idea? Thanks
Reply

Cezar Santos
April 15, 2011 at 9:28 AM
This post was build on 11g but I dont see problems in apply to 10g. Maybe some smalls details
To drop the J$ and JV$ use the stop journal option from model or table.
Make sense?
Reply

Leave a Reply
Required fields are marked *.
Message *

Name *
Email *
Website

Confirm you are NOT a spammer

Previous Post
Next Post

Recent Posts
ODI Expert and why there is no new posts
Passing Values to ODI Variables in Packages/Scenarios
New Author Arriving
ODI Certification Approved
How to Change ODI Language (Idiom)

Recent Comments
dba on How to use PL/SQL procedures and functions in
ODI
Krishna on IS_NUMBER (and IS_CHAR) at Oracle? A
workaround
Gagan on Export and Importing Scenarios based on Project
(or)Folder
Raghabendra Sinha on IS_NUMBER (and IS_CHAR) at
Oracle? A workaround
Ranjeet Srivastav on ODI Expert and why there is no new
posts

Archives
February 2016
April 2015
March 2015
February 2014
November 2013
September 2013
December 2011
November 2011
September 2011
August 2011
June 2011
March 2011
February 2011
January 2011

December 2010
November 2010
October 2010
September 2010
August 2010
July 2010
June 2010
April 2010
March 2010
January 2010
December 2009
November 2009
October 2009
September 2009
August 2009

Categories
Administration
Architecture
Common Errors
Drivers
How to
Java Codes
Knowledge Modules
Logic
ODI
ODI Bug
Others
Reports
SDK
SQL
Technology
Tips and Tricks
Uncategorized

Meta
Log in
Entries RSS
Comments RSS
WordPress.org

Recent Jobs
ODI Expert and why there is no new posts
Passing Values to ODI Variables in Packages/Scenarios
New Author Arriving
ODI Certification Approved
How to Change ODI Language (Idiom)
Information about the ODI Certification Exam
Finally!!!! The Oracle Data Integrator 11g Certified
Implementation Exam!!!!!!
Interface with Union, Minus Operator using ODI SDK
Export and Importing Scenarios based on Project
(or)Folder
Creating Interface for Single Source and Target
Creating Permanent Interface based on Model Level
Automatically add Interfaces/Scenarios to new
ODIPackage using ODI SDK
Creating Package using SDK
Handling the ORDER BY clause in an interface

Creating Temporary Interface using ODI SDK

Creating ODI Procedure using SDK
ODI SDK Setup and Config in Eclipse
Things to be pride of.
CDC CONSISTENT
Ordered SQL
Killing session in ODI 11g
11g Oracle Data Integrator Part 8/11g Table
Partitioning supported
Jython Scripts fails to recognize the module during
scenario run in ODI 11g
Scenarios, a lot in the first delivery makes maintenance
very simple!
Direct load by interface with no work tables (C$, I$)
11g Oracle Data Integrator Part 7/11g SUNOPSIS
MEMORY ENGINE
11g Oracle Data Integrator Part 6/11g Union , Minus,
Intersect
Jythons Beginner Guide for ODI
Send SQL by Email
Security Module Main concepts and use
Drivers Location in ODI 11g
11g Oracle Data Integrator Standalone Agent but, and
about the Windows Service???
11g Oracle Data Integrator Part 5/11g odiConsole, the
new Metada Navigator
Using one of ODIs lesser-known secrets: the Common
Format Designer (CFD) by Craig Stewart
11g Oracle Data Integrator Part 4/11g Upgrading from
10g
11g Oracle Data Integrator Upgrading from 10g Part
4/11g
First year of many. Thank you Friends!
11g Oracle Data Integrator Part 3/11g Graphical User
Interface
Datatypes Creating what is missing for any technology
IS_NUMBER (and IS_CHAR) at Oracle? A workaround
Calling OdiSqlUnload using ODI Procedure with no
hardcoded password
SNP_EXP_TXT A huge table problem
Internal ID
SCD Type 2
11g Oracle Data Integrator Part 2/11g Creating
Repositories
11g Oracle Data Integrator Creating Repositories Part
2/11g
Regenerate ODI Objects using OdiGenerateAllScen
Generate Column_name Header for OdiSqlUnload
Error Records Log
Get the Error Message of any Failed ODI objects inside
Package
SCD TYPE 3
Best practice of ODI
11g Oracle Data Integrator Part 1/11g First Screen
Automated ODI Export Import Scenarios
LKM OdiSqlUnload(File) to Oracle ( SQLLDR )
Jython OS bug in Windows 7
Writing SQL counts using Jython
Getting one or several unknown files from a directory
Interface Parallel Execution A new solution
Displaying SQL results in Operator
Flow Control and Static Control
Export DWR to EWR [ Common Master Rep ]

Scenario Report
Asynchronous -Parallel Execution
Dropping $ Tables
KM FOR UNION BASED ON IKM SQL CNTRL
APPEND
Installing ODI in Windows 7
Querying in ODI
Changed Data Capture ( CDC )
Agent
Manual entry in ODI Driver List
MINUS IN ODI
Variable Detail Query
Interface Mapping Query
Topology Connection Detail
Versioning
Get File Length and Header in Operator
Find the Value of the Variable in Operator ?
Invisible Join
Shortcuts Keys
Export Import DWR to EWR having same Master Rep
Categorize Scenarios
Alignment of objects in Package
Duplicate Multiple Objects in Package
Smart Search
Export Import DWR to DWR having same Master
Repository
Procedures (or KM) Steps Source and Target tabs
Subscribers An example of how to use
Why Creating a separate schema in ODI?
How to show Business Messages on Operator and, too,
debug variables
Topology Data Server Which user should be used to
connect?
How to use PL/SQL procedures and functions in ODI
Context, Logical And Physical Schema How does it
work?
Repository Architecture Work Repository, Development
or Execution? When and why uses each one.
Removing Special character using Jython script
Faster and Easiest way to design Interfaces
AUTOMIZE ODI REPOSITORY EXPORTS
Informix Connection
SQL Server connection
Repository Architecture Two or more Masters Part 3
Report of list of KMs in each Interface
Connecting openLDAP
How to call a Web Service that start an ODI Scenario and
waits its complete execution
Processing multiple Interface through single Package
Multiple Files Single Interface
Generate Multiple Scenarios
Sequence
How to get a constant or pseudo-column from source
mapping
Lock Windows Layout
Hide Unused Technologies
Tab at the Top
oracle.jdbc.V8Compatible
Oracle to Flat file
Scheduling using ODI Scheduler
Exception getobjectname schema
How to create a Temp Table in ODI Interface (Yellow
Interface)

Printing report
Step by step procedure to Read Excel (*.xls)
Importing Knowledge Modules
Creating Master and Work Repository
Installation
Repository Architecture Just one Master Part 2
How to refresh ODI variables from file Part 2 Getting
all lines, once at time
How to refresh ODI variables from file Part 1 Just one
value
Repository Architecture Just one Master Part 1
Welcome to ODI Experts, this is Cezar Santos.

Copyright 2016 | All Rights Reserved

Scroll