Copyright 2010, Oracle. All rights reserved.

PTS Workshop: ODI 11g

<Insert Picture Here>

Oracle Data Integrator Model Concepts

Agenda
Model Overview Reverse Engineering Model Creation Complex File Driver Lab Overview Q&A
<Insert Picture Here>

Copyright 2010, Oracle. All rights reserved.

Model Overview

Copyright 2010, Oracle. All rights reserved.

What Is a Model?
Models are the objects that will store the metadata in ODI.

They contain a description of a relational data model. It is a group of datastores stored in a given schema on a given technology.
A model typically contains metadata reverse-engineered from the real data model (Database, flat file, XML file, Cobol Copybook, LDAP structure) Database models can be designed in ODI. The appropriate DDLs can then be generated by ODI for all necessary environments (development, QA, production)

Copyright 2010, Oracle. All rights reserved.

Relational Model
ODI is strongly based on the relational paradigm. In ODI, data is handled through tabular structures defined as data stores. Data stores are used for all types of real data structures, including flat files, Extensible Markup Language (XML) files, and Java Message Service (JMS) messages.

Copyright 2010, Oracle. All rights reserved.

Relational Model Support in ODI

All elements of relational models are described in ODI metadata:
Relational model
Table; Column Not Null; Default value Primary keys; Alternate Keys Indexes; Unique Indexes Foreign Key Check constraint

Description in ODI
Data store; Column Not Null/Mandatory; Default value Primary keys; Alternate keys Not unique indexes; Alternate keys Reference Condition

Copyright 2010, Oracle. All rights reserved.

Additional Metadata in ODI

Filters: Apply when data is loaded from a data store Heterogeneous references: Link data stores from different models or technologies Additional optional technical or functional metadata: Online analytical processing (OLAP) type on data stores Slowly changing dimension behavior on columns Read-only data types or columns User-defined metadata (FlexFields)

Copyright 2010, Oracle. All rights reserved.

Reverse Engineering

Copyright 2010, Oracle. All rights reserved.

What Is Reverse Engineering?

Reverse engineering is an automated process to retrieve metadata to create or update a model in ODI.

Copyright 2010, Oracle. All rights reserved.

Two Methods for Reverse Engineering

Standard reverse-engineering
Uses JDBC connectivity features to retrieve metadata, then writes it to the ODI repository. Requires a suitable driver

Customized reverse-engineering
Read metadata from the application/database system repository, then writes these metadata in the ODI repository Uses a technology-specific strategy, implemented in a Reverseengineering Knowledge Module (RKM)

Copyright 2010, Oracle. All rights reserved.

Standard vs. Customized ReverseEngineering

File-specific reverse-engineering
Fixed format
COBOL copybooks

ODI Repository
Model (Metadata)

Oracle Data Integrator

Delimited format

MS SQL Server

JDBC Driver
Standard Reverse-engineering

Data Model

System tables

Customized Reverse-engineering

Copyright 2010, Oracle. All rights reserved.

Other Methods for Reverse-Engineering

Delimited format reverse-engineering
File parsing built into ODI.

Fixed format reverse-engineering

Graphical wizard, or through COBOL copybook for Mainframe files.

XML file reverse-engineering (Standard)

Uses Sunopsis JDBC driver for XML.

LDAP directory reverse-engineering (Standard)

Uses Sunopsis JDBC driver for LDAP.

Copyright 2010, Oracle. All rights reserved.

Note
Reverse engineering is incremental. New metadata is added, but old metadata is not removed.

Copyright 2010, Oracle. All rights reserved.

Model Creation

Copyright 2010, Oracle. All rights reserved.

How to Create a Model by Reverse Engineering

1. Create an empty model. 2. Set up the reverse engineering strategy. Standard versus customized reverse engineering 3. Run the reverse engineering process. Use the selective reverse option (standard reverse engineering). 4. Flesh out the model. Add data stores, columns, constraints, and so on.

Copyright 2010, Oracle. All rights reserved.

Step 1: Creating and Naming a New Model

1. 2. 3. 4. 5. Select the Models view. Click the New Model icon. Enter the name (and code). Select the model technology. Select the logical schema where the model is found. 6. Enter the description (optional).

Copyright 2010, Oracle. All rights reserved.

Note
A model is always defined in a given technology. If you change a models technology, you must recheck every object related to that model.

Copyright 2010, Oracle. All rights reserved.

Step 2: Defining a Reverse Engineering Strategy

1. Click the Reverse tab. 2. Select the reverse engineering type. 3. Select the context for reverse engineering. 4. Select the object type (optional). 5. Enter the object name mask and characters to remove for the table alias (optional). 6. If customized:
Select the RKM Select the Logical Agent

Copyright 2010, Oracle. All rights reserved.

Step 3: Starting the Reverse Engineering Process

Right-click the model and select Reverse Engineer

Copyright 2010, Oracle. All rights reserved.

Customized Reverse Engineering

3 1

2
3

Copyright 2010, Oracle. All rights reserved.

Selective Reverse Engineering

1. 2. 3. 4. 5. 6. Click the Selective Reverse-Engineering tab. Select the Selective Reverse-Engineering check box. Select the New Datastores and/or Existing Datastores check box. Click the objects to reverse. Select the data stores to reverse engineer. Click the Reverse button.

Copyright 2010, Oracle. All rights reserved.

Add Elements Missing From Models

Some metadata cannot be reverse-engineered
JDBC driver limitations

Some metadata cannot exist in the data servers

No constraints or keys on files or JMS messages Heterogeneous joins OLAP, SCD, etc.. User-defined metadata

Some business rules are not implemented in the data servers.

Models implemented with no constraints Certain constraints are implemented only at the application level.

Copyright 2010, Oracle. All rights reserved.

Step 4: Fleshing Out Models

ODI enables you to add, remove, or edit any model element manually. You do this in Designer Navigator. The model diagram is a graphical tool for editing models. You can update the database with your changes.

Copyright 2010, Oracle. All rights reserved.

Complex File Driver

Copyright 2010, Oracle. All rights reserved.

Complex Files
Introduction
New technology Driver parameters KMs to use

Supported/non-supported
Compare to XML Insert graphic C.4.

26

Complex Files
Introduction New technology Concepts and processes similar to XML files
Main difference: Step that transparently translates the Native File into an XML structure that is used internally by the driver but never persisted

Complex (or native) File format

Native Schema (nXSD) file describes file structure

A Complex File corresponds to an ODI data server

Within this data server, a single schema maps the content of the complex file

nXSD file + Complex Files driver ODI is able to reverseengineer, read, and write information from complex files
27

Complex File Processing

Seller
6335722^Company One^First Street 999 San Jose 95129USCA650-801-6250

^Oracle^Bridge Parkway 1600

Redwood Shores 94065USCA650-506-7000

001|Oracle Data Intagrator EE|20000,2,+40000+ 002|Application Adapter for ODI|10000,5,+50000+ 003|BPEL Process Manager Enterprise Edition|20000,2,+40000+ 004|BPEL Process Manager Standard Edition|10000,5,+50000+ 005|BPEL Process Manager Developer Edition|1000,20,+20000+#110000

Purchaser Line Items

<schema attributeFormDefault="qualified" ... <element name="invoice" type="tns:invoiceType" /> <complexType name="invoiceType">

<sequence>
<element name="purchaser" type="tns:partnerType" /> <element name="seller" type="tns:partnerType" /> <element name="line-item" type="tns:line-itemType" ... </sequence> Complex File Schema (nXSD) generated with Native Format Builder Wizard. </complexType>
...

Complex File Driver

Supports Delimited, Fixed, Complex files, Cobol Copybooks.

28

ODI Driver for Complex Files

Overview

ODI
Complex File Driver

XML to

relational

XML

Complex File

XML nXSD
29

ODI Driver for Complex Files

Parameters
JDBC Driver:
oracle.odi.jdbc.driver.file.complex.ComplexFileDriver

JDBC URL:
jdbc:snps:complexfile?f=<native file location>&d=<native schema>&re=<root element name>&s=<schema name>[&<property>=<value>...]
Property
f

Value
<native file name>

Description
Native file location (mandatory)
Use slash "/" in the path name instead of back slash "\" Possible to use an HTTP, FTP or File URL (Files located by URL are read-only.)

d re

<native schema> <root element>

Native Schema (nXSD) file location (mandatory) Name of the root table element of the schema (mandatory) Case sensitive Used for reverse-engineering

<schema name>

Relational schema name where the complex file is loaded (mandatory) Select when creating the phys. schema under the Complex File data server

30

ODI Driver for Complex Files

URL Example
jdbc:snps:complexfile?f=/infiles/PROD20100125_001.csv &d=/infiles/products.nxsd&re=root&s=PRODUCTS

Connects to the PROD20100125_001.csv file described by products.nxsd and exposes this file as a relational structure in the PRODUCTS schema

31

Lab Overview

Copyright 2010, Oracle. All rights reserved.

Lab 5: Reverse Engineering into Models Overview

Target Data sources
Model
Logical schema: TRAINING_SRC

Model

Logical schema: ORACLE_ORCL_LOCAL_SALES

Oracle Sales Application

MYSQL_SRC Predefined
SPECIFIC_SEQ, SRC_CITY, SRC_CUSTOMER, SRC_ORDERS, SRC_ORDER_FORM, You will reverse SRC_ORDER_LINES, SRC_PRODUCT, engineer SRC_REGION

You will reverse engineer

Datastores (tables) of target model

TRG_COUNTRY

TRG_PRODUCT

TRG_SALES

TRG_REGION

TRG_PROD_FAMILY

TRG_CUSTOMER

TRG_CITY

Copyright 2010, Oracle. All rights reserved.

Copyright 2010, Oracle. All rights reserved.

Copyright 2010, Oracle. All rights reserved.