Oracle Data Integrator

Oracle Data Integrator
Class-room training

Feb 2014
Copyright 2013 Deloitte Development LLC. All rights reserved. 2
Data Integration- Scenario 1
Data
Transformation
Staging Area
Oracle ERP
PeopleSoft
SAP
Data Warehouse
Essbase

Heterogeneous Source Databases
spread across geographies. Data will be
Extracted from its tables

Select * from Source Tables where C1 Cn
Transformation Engine will
remove semantic differences and
apply filter conditions C1 through Cn

Transformed data is Inserted (Loaded) into
Target database which is further used for
Analysis using BI applications

Data Integration- Scenario 2
Data
Transformation
Staging Area
JD Edwards
Oracle ASCP

Source Database is an ERP application
which has information related to
Sales Orders
Receipt and Routing
Transfer Orders
Bill Of Materials
Calendars

Target Application uses specific ERP data
for Supply chain Planning. Data from Source
tables is inserted into specific target tables on which
Supply chain planning algorithms are applied
Data transformation involves changing Date Format, Order and
Receipt codes, additional information which is derived by
appending specific source columns etc.
Challenges of Data Integration
Transformation Engine :
Should be able to transform large volumes of data without hampering integration time
Should have enough memory for processing large volumes of data
Should support different SQL and non-SQL languages. E.g. Source can be a Relational
table (SQL), a flat file (JDBC), excel sheet (ODBC), XML or specific technology like
Essbase (MDX)
Should be able to handle semantic differences of SQL technologies
Should be able to handle Change data capture (CDC) and Journalizing
Should perform referential and conditional checks
Should be scalable
Should have minimal learning curve and implementation time
Oracle Data Integrator
ODI uses E-LT architecture which leverages SQL capabilities of Source or
Target systems apart from having its own Staging Area
This technique reduces dependency on staging area since transformations can
be done at source or target
Declarative approach for development, reduces dependency on understanding
technology specific syntax and semantics

ODI Basic Concepts
Main Window before connecting to a
Repository
ODI has four important navigation panes
1. Designer
2. Topology
3. Operator
4. Security
ODI Repositories - Introduction

User Login Info
Source/Target
server information
Technology
information
Agent Information
Interfaces
Models
Projects
Knowledge
Modules
Interfaces
Models
Projects
Knowledge
Modules
Repositories are a set of tables in a Database schema which
store ODI related information which is manifested graphically in
ODI Studio
ODI has 2 Repositories : Master and Work
Each Installation will have only one Master Repository and can
have more than one Work Repositories
A set of Master and a Work Repository gives a connection
Setting up Repositories Master Repository

1. To create a new repository, Click on New

2. Select Master Repository Creation
Wizard to create a new Master
Repository

Option: To import an already existing
Repository, select this option
(used in migration from env to env)
Enter the following details as per the
Database, for master repository creation
DBA user credentials are required
Enter the DBA user and its password.
Valid users ( Sys or System)
Schema should have connect, resource
privileges
ODI Authentication creates and manages
user information within ODI.
For using External Authentication like
Enterprise login credentials select the other
option
Choose Internal Password Storage and click on Finish
button. External password storage can be used
when ODI is installed on top of a Application Server.
1.Click on Connect To Repository, a login window
should open
2.Click on the button to create a Login. Provide
the Connection Information: User
SUPERVISOR and its password, enter the
Master Repository details that was
created, select Master Repository Only option
Setting up Repositories Work Repository
Go to Topology navigator pane, in Repositories
create a new Work Repository by right-clicking
on the section
Note : Work Repository can be a new schema
or the schema used for Master Repository. However
it is a good practice to keep Work and Master
Repositories in different schemas. Work repository
Schema and Master repository Schema should be
given Connect, Resource privileges.

Enter details, test connection and click on Next
Work Repositories are of two types
Development : Used when developing integrations
Execution : Used in production environment where
Interfaces are only executed. No development can
be done in this mode. Designer Pane will be disabled
Password can be set if Work Repository needs to be password
Protected.
Select Development type and press Finish
1. Click on Edit button for previous login
2. Select Work Repository and click on
the select button
3. Select the repository
and press OK
Starting the development
- Creating Topology
ODI Development stages
Create Topology
Create Models
Develop Interfaces
Automate
Integration
Topology Manager
Create Physical Schema
Create Context
Map Physical Schema to logical schema
Desginer
Import Knowledge Modules
Create Models, reverse tables
Designer
Create Variables
Create Interfaces, procedures
Create Packages and Scenarios

Operator
Import Scenarios
Create batch scripts to execute scenarios
Tell ODI where Source, target and staging
Databases are located

Tell ODI which technology codes it needs
to use (Java for file, SQL for SQL, MDX for
Essbase etc.)
Show source, target tables graphically in
ODI

Create graphical mappings of source and target
Tables to complete the integration

Create Scenarios, automate execution using batch
Scripts

Topology Manager - Introduction
Physical Location of Sources and Targets are set in
Topology. Information such as Server name, port, JDBC
Driver, JDBC URL, Schema/Username and Password
are provided within Physical schema
Context is used to map a
Physical schema to Logical name
(explained in later section)
Work Repositories are created
here
Creating Topology Physical Schema
Physical Schema in ODI means the actual
location of schemas which will be used for
integration either as source or target or
staging
Data Servers in ODI denote the actual
location of Database servers which will
contain db schemas
Right Click on the technology and select New
Data server to create a new data server. In this
case our Data server is a File system
Specify a suitable name
In the JDBC Section, select the Driver
By clicking on the Search button
Note: URLs sometimes need optional
Information which is specified in square
Braces [], delete them when not required
JDBC drivers are used by ODI to communicate with Databases, in some cases JDBC
drivers are not available e.g. Hyperion Essbase/Planning technology will not have
JDBC drivers
Once Data servers are created, you need to create
a physical schema, schema is the physical location
of files in this case
Exact location of folders need to be specified, for
Shared folders, location can be specified as
\\serverName\FolderName
Windows for Physical schema, Data Servers are generic, hence some technologies will have some
functionalities disabled
Creating Data Server for Oracle Technology
Give a suitable name,leave the instance/dblink
blank as its not required for Oracle technology
(Generic windows for all technologies)
Create a new physical schema
If Data Server details are correct, Schema will be
automatically displayed
Work Schema: All temporary tables
that ODI will create for transformation,
error checks, journalizing will be created
in this schema within the database.
Helps in optimization of resources.
Creating Topology Context and Logical Schema
Physical Locations Contexts
Logical Name
Contexts help in mapping different physical locations to one logical location.
E.g.
In the example below, Server1 and Server2 are having identical DBs and
schema which will be used in case one dataserver fails (failover
mechanism).
ODI_TRAINING is mapped to two physical locations
When designing or executing integrations, locations will always be referred
using logical name (ODI_TRAINING) and the context will specify which
physical location to connect
Creating Topology Context and Logical Schema
E.g. When creating a model, schema is referred using
Logical names, note that physical locations are never
Referred in ODI while developing objects
In the next stage, we specify the context to be used
Logical Name + Context together
identify a particular Data Server and
Physical Schema.
Contexts are widely used to have
multiple environments to be worked
upon e.g. Production, Development,Test
Starting the development
- Importing Knowledge Modules
- Creating Models
Designer - Introduction

Project is a container which holds all integration
Interfaces, Packages, KMs, Variables and Procedures
No objects can be shared amongst two projects,
Objects can be shared within a project and logically
separated using folders

Model point to a Schema, they are reversed-engineered
to get the DataStores.
Reversed-engineering is a process of ODI connecting
to the Information Systems (Sources and Targets) and
retrieving the structures of the objects (Tables/Views)
of that Information System into ODI.
Datastores are graphical representations of
Tables/Views within a schema (Model in ODI )
Operator - Introduction
Operator window is where logs of
execution are created
These logs can be exported to an
XML file
Apart from maintaining logs,
Operator window is important in
Production environment since
scenarios are imported and
executed from this window
Creating Project
Project is a logical grouping of Integration objects.
Integration Objects are
1. Interfaces
2. Packages
3. Variables
4. Procedures
5. Knowledge Modules

Creating Project
Click here to create a new project
Save
Knowledge Modules
ODI follows a declarative approach of development (programming).
This approach emphasizes on what to do rather than how to do. This
translates into less of coding in ODI and more of graphical representation of
programming flows.
Table 1
Column1
Column2
Column3

...
ColumnN
Table 2
Column1
Column2
Column3

...
ColumnN
Procedural approach, developer has to
write code, which results in syntactical
errors, moreover developer needs to have
knowledge of semantics and syntax
Declarative approach,
developer will define the
Flow graphically, code
will be generated
automatically behind
the scenes
Knowledge Modules
Knowledge modules are templates of code that define the integration. They are
built as per the data-warehousing best practices.
Graphical representation created by developer
Code ODI creates in the back-end with KMs
Knowledge Modules
KMs are classified into six types
Reverse KM (RKM)
Load KM (LKM)
Integration KM (IKM)
Check KM (CKM)
Journalizing KM (JKM)
Service KM (SKM)

Purpose of using KMs
Less learning curve, since it is not necessary to learn syntax/semantics of
various SQL/non-SQL technologies
Development and Implementation time is reduced no code debugging
required
Knowledge Modules
RKM (Reverse Knowledge Modules) are used to perform a customized reverse-
engineering of data models for a specific technology.
LKM (Loading Knowledge Modules) are used to extract data from one or more source
datastores (files, db table, excel, xml, mainframe, etc.).
JKM (Journalizing Knowledge Modules) are used to create a journal of data
modifications (insert, update and delete) of the source datastores to keep track of the
changes.
IKM (Integration Knowledge Modules) are used to integrate (load) data to the target
datastore.
CKM (Check Knowledge Modules) are used to check that the constraints on the targets
are not violated.
SKM (Service Knowledge Modules) are used to generate the code required for creating
data services.
Knowledge Modules
Table 1
Column1
Column2
Column3

...
ColumnN
Table 2
Column1
Column2
Column3

...
ColumnN
Reverse KM will extract the
object/table structures into ODI
Essentially execute Desc table
query on SQL tables
Load KMs will extract data from
Source tables
(Execute select queries)
Integration KM will load data into
Target tables,
Execute insert into table query
Uses of RKM, LKM and IKM
Table 1
Column1
Column2
Column3

...
ColumnN
Table 2
Column1
Column2
Column3

...
ColumnN
Importing KMs
1. Right click on Knowledge Modules and then click on
Import Knowledge Modules to import them in ODI
2. Select desired KMs and click on OK
The File Import Directory is auto selected /picked-up
Once KMs are imported, they are
visible as nodes in KM tree
(Open them to view the code)
Creating Models
Create a new model
Models are graphical representations of database schemas. Within the
models, tables are graphically represented and are called DataStores.
Models are reverse-engineered to get the DataStores.
Reverse-engineer is a process of ODI connecting to the Information Systems
and retrieving the structures of the objects of that IS into ODI.

Creating Model Oracle
1. In Definition section, give a desired
Name, Code is generated automatically
Select technology Oracle, and desired
Logical Schema, ignore all other fields
2. For Oracle Technology, Reverse
Engineer tab is set to standard.
Select type of objects to reverse
(tables, views, synonyms etc).
Go to next tab
Creating Model Oracle

1. Check Selective Reverse Engineering then New
Datastores and Objects to Reverse engineer. A list of all
tables within the schema are displayed, check the ones
that are needed to be reversed and click on
Reverse Engineer

2. List of all reversed tables/views are displayed under the model,
double click on them to see the columns and other information of
those tables. This is how source and target tables are drawn
In ODI. These are called as Data Stores in ODI
Creating Model File
Data Stores of a File model need to be created
manually (ODI wont reverse a folder automatically to
show the files as Datastores)
E.g. A text file EMPDEPT.txt is a text file that will be
used as a datastore in ODI.
Note the location of file is specified in Physical
schema as explained earlier.
A model of File technology is created and then
select New Datastore option by right-clicking on
the file model that was created.

Creating Model File
1. Give a suitable name for the datastore and
click on the search button for Resource Name;
a browser window pointing to the folder
specified in the physical schema is displayed
Note: You cannot select a file that is out of
the folder mentioned in Physical Schema.
2. Select the file and click on Open.
Creating Model File
1. Flat file properties need to be selected
Properly to avoid issues in reversing.
In this example, the flat file is comma
Delimited with first line as header and
has no text delimiter or decimal separator
The format is MS-DOS (decides line-feed)
character.
Proceed to Columns tab after specifying
details of flat file.
2. Click on Reverse engineer button.
If details entered in previous tab are correct,
A list of column will be generated.

Creating Model File
Flat-files are reversed as a datastores into ODI.
Note: User can change the length and scale of columns in this window
(applicable to flat files/XML only)
Additional columns can be added by clicking on the add button. But make
sure that the same columns are added to the flat file using a file editor.
(notepad, vi editor etc).

Creating Interfaces
1. Right Click on Interface node and click on
New Interface
2. In the Overview Tab, the first section is
Definition, Enter a suitable name.
3. Provide description of the Interface
Note: Staging area different from Target will be enabled when a Target is specified in Mapping tab.
Developer needs to revisit the Overview tab again after specifying the target
Creating Interface
In the Mapping Tab, drag and drop source
datastores in the Source area as shown.
Developer can click on the datastore and have
look at the Property Inspector. Context is
<Execution Context> meaning, user will specify
the context when executing the interface
Creating Interface

1. Similarly, drag and drop the Target datastore
in the Target Area, Automatic Mapping option will
auto-map the columns with the same name

2. Click on the Target Datastore Header to view
the properties in Property Inspector
Note: when Target Datastore is set, the Staging Area different
from Target option is enabled in the Overview Tab
Drag and drop DEPTNO from one table to DEPTNO of the other table,
this will create a join between the two sources, A red cross mark shows
an error (ODI does this check using the Optimization context). Click on
button to view errors.

Go to Overview tab and choose Optimization context as Local. The error
will be removed.
Notice that Staging area different from Target is enabled. Selecting this
will not use Target schema for staging (creating Temporary Tables for
transformation )
In the Interface, the target is a Flat file which does not have SQL
capabilities for transformation, hence selecting this option is necessary
in this case.
Creating Interface
Click on the join icon , the
Property inspector will show the
implementation of join
Two radio buttons next to Execute
on show where the join is
implemented. If Source is selected
join will be performed while
extracting from Source into one
temporary table, Staging Area will
create multiple temporary tables,
one for each source tables. This
helps in optimizing memory usage
of Source, Target and Staging
schemas.
Creating Interface
Creating Interface
Drag Column names from Source tables into
the Mapping section of Target Table. Click on
a particular Column of Target table to view its
properties. Mapping can be selected to be
executed on Source or Staging or Target
Schema
Creating Interface
In the flow tab, Click on the Target Area, Target
Properties are now displayed in the Property
Inspector, go to IKM selector and select the
Particular IKM required for this technology
(KM names are self explanatory, and will be
visible only if they are Imported prior to creating
interface)
Note: Since Our Source and
Staging area Schemas are
same we do not required LKM
but if Source and Staging are
different then LKM need to be
selected too. (next slide)
Creating Interface
When Staging and Source are two different
Physical schemas, LKM will be needed. Click
on the top of the Source to select the LKM as
Shown below. (Assuming KMs are imported)
Create Interface
Parameters can be set to the
apporpriate option in a KM
Quick Edit Tab gives a
summary Of the Integration
Executing an Interface
Click on the Execute Button below the
Menu bar after saving the Interface.
A dialogue box will show the parameters to
be selected
Context: Will select the actual physical
schemas on which the interface is
executed
Agent : Port on which java code is
executed
Log Level: specifies level of detail in log
files
Simulation: if checked, it will not actually
run the Interface, but just generates the
code execute to check any syntax errors
Go to the operator window to check the
status
Creating Interface Adding Filters
Filters are selection criteria that are applied on the
source tables in order to select a particular set of
data for integration
Filters are created by dragging and dropping the
column in the empty space of Source Area in
Mapping tab.
Click on the filter icon to view its properties
In the implementation section, filter can be
Implemented, or click on Edit button to use
SQL functions and keywords
Creating Interface Applying Filters
SQL Code applied. Drag and drop Data
Sources (to the left), Variables and
Functions (at the bottom) in this area.
Only functions/Keywords of the technology
on which the execution is selected are
available.
E.g. if filter is executed on Source schema
which is MS SQL server, then functions/
Keywords supported only in MS SQL will
work. Oracle SQL functions will not work

Creating Variables
Give a suitable name and specify the datatype in the Definition
Section. Keep History will keep a track of values the variable is
holding so far. Select the option needed (Options are self
explanatory) . Secure value will encrypt the value.
Default value is used when a Variable is used outside of Package
ODI variables are placeholders for keeping data that can be
reused. They are referred using a prefix #
(e.g. #VAR_DEPTNO)
They can be:
1. Alphanumeric
2. Number
3. Date
4. Text
ODI can also use database level variables but these
variables are stored in ODI repositories.
Right click on the Variables node to create a new variable
After the query is declared, click on the Refresh button to populate the
variable. A new session is started, check if the execution is successful in
Operator window.
When the execution is successful, go to History tab to view the contents of the
Variable. (Depending on the Keep History option, values will be populated)

Creating Variables
Refresh tab will specify how the variable is populated.
In this case, the variable will hold the count of DEPT
Table, developer can use the edit button to create the
query.
Creating Variables
Here, it is demonstrated how one variable can be used
to store value of another ODI variable
One more variable called DeptRowCountHist is created.
This variable will be used to store the value of
DeptRowCount variable created earlier
Another variable is referred by prefixing # and Project
Name to the variable. This is necessary since the
Scope of variable needs to be defined while setting
the value of it. Also note the table name is dual
Creating Package
Package is a set of objects arranged in a logical flow of their execution sequence.
Interfaces, procedures, variable validations are joined to form a flow of execution
sequence.
They are created by right-clicking on the Package node. Notice that there are two tabs
(Overview and Diagram) at the bottom. Details are provided in the Overview tab as
shown
Creating Package
Drag and drop variable from Project
section to the right In the empty space of
Package. Green arrow sign means this is
the starting point of the package
When the object is selected, the properties window
Shows the action to be performed. For a variable the
Options are Refresh Variable, Set Variable and Evaluate
Select Refresh Variable
Creating Package
Likewise, drop the variables shown, the third step is comparing the
Variable DeptRowCount with DeptRowCount Hist. Notice the execution
is set to Evaluate variable and it is evaluated against DeptRowCountHist.
Variable is referred using # and Project Name as prefix
Logic: DeptRowCountHist will store the previous count of rows of DEPT table,
DeptRowCount will store the latest row count of DEPT table. If values match
an Interface to populate EMPDEPT.txt table should execute, else no execution
Creating Package
Interface is dragged and dropped. One more variable named execution status is created and dropped
in the diagram area. Using the OK and KO arrows, drag them from one object to another to create a
flow as shown below
Once all objects are dropped in Diagram pane, a flow needs to be created
Generating Scenarios
Scenario is similar to a compiled java code which is stored in an executable format. Scenarios can be
Executed from a batch file (merged with other technologies batch scripts)
Exported to a production environment for execution purpose only
Scheduled to be executed at a particular time by using ODI scheduler or OS Batch schedulers
1. Right-click on the package created and click
on Generate scenario
2. Default name and version is populated. Click on OK

Oracle Data Integrator

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Oracle Data Integrator

Hochgeladen von

Copyright:

Verfügbare Formate

Oracle Data Integrator

Das könnte Ihnen auch gefallen