ETL Process Optimization using Push Down

Optimization (PDO) and Teradata Parallel

Transporter (TPT)

Author: S Anand
Project: GE Capital Fleet Management

Introduction
Overview of Pushdown Optimization (PDO)
How does PDO Works
How does Integration Service handles PDO
Explanation with 2 examples
Types of PDO
Active and Idle Databases
Working with Dates
Rules and guidelines for functions in PDO
Error handling, Logging and Recovery
Configuring sessions for PDO
Limitations on types of transformations that can be pushed to DB
Benefits of PDO
Alternative to PDO- Teradata parallel transporter (TPT)
What is TPT and Types of Operators

Overview of Pushdown Optimization (PDO)

Load balancing among servers
Push the ETL logic to Database

How does PDO Works

Push transformation logic to Database
Translates logic in to SQL queries and sends to Database
Push transformation logic to Database

How does Integration Service handles PDO

Creation of Temporary objects
Temporary Sequence objects and temporary views
Session property- Push down optimization with sequence/View

Home

Example 1

Push Down Query

INSERT INTO T_ITEMS (ITEM_ID, ITEM_NAME, ITEM_DESC) SELECT CAST ((CASE WHEN
5419 IS NULL THEN '' ELSE 5419 END) + '_' + (CASE WHEN ITEMS.ITEM_ID IS NULL THEN ''
ELSE ITEMS.ITEM_ID END) AS INTEGER), ITEMS.ITEM_NAME, ITEMS.ITEM_DESC FROM
ITEMS2 ITEMS.

Example 2 ( Filter transformation with condition DEPTNO>40 )

Push Down Query

insert into emp_tgt(empno, ename, sal, comm, deptno)
select
emp_src.empno,
emp_src.ename,
emp_src.sal,
emp_src.comm,
emp_src.deptno
from emp_src
where (emp_src.deptno >40)

Push Down Optimization Viewer

Home

Types of Pushdown Optimization

Source Push Down Optimization
Target Push Down Optimization
Full Push Down Optimization

Integration Service behaviour with Full Push Down Optimization

Home

Working with Dates

Date values converted to character values

Date formats for TO_CHAR and TO_DATE functions\
HH24 date format
Blank spaces in date format strings
SYSDATE built in variable

Rules and guidelines for functions in PDO

If you use ADD_TO_DATE in transformation logic to change days, hours, minutes, or
seconds, you cannot push the function to a Teradata database
When you push LTRIM, RTRIM, or SOUNDEX to a database, the database treats the
argument (' ') as NULL, but the Integration Service treats the argument (' ') as
spaces.

Home

Configuring Session for PDO

Home

Limitations on type of transformations

Home

Benefits of PDO
The Power Center Pushdown Optimization Option offers many benefits, including:
Increased performance by using optimal resources
Increased ease-of-use with a metadata-driven architecture that provides metadata
lineage
Increased IT team productivity with simplified debugging and performance tuning
Reduced risk and enhanced flexibility through database neutrality
Conclusion
Despite being successful we had tough time in implementing it because of the Rigid
Rules and Guidelines to be followed for the PDO to work and also this technique
was not feasible for a few of our mappings which handled transformations /
Circumstances that do not support PDO .In such case we looked for options other
than PDO. However Pushdown optimization proved to be successful method of
improving performance of jobs

Home

Teradata parallel Transporter

Teradata Parallel Transporter (TPT) is a parallel multi-function load environment. It
was designed to use the different Teradata Load and Unload Utilities under a single
infrastructure. So, based on the protocols of fastload, multiload, tpump and
fastexport, new "operators" were developed and they are named as Load, Update,
Stream and Export respectively
Teradata parallel Transporter Operators
Stream
Update
Load