White Paper

BusinessObjects Data Integrator

Pushdown SQL Functionality

Maximizing The Number Of Pushed Down Operations

To optimize performance, Data Integrator pushes down as many transform operations as possible to the source or target database and combines as many operations as possible into one request to the database. Data flow design influences the number of operations that Data Integrator can push to the database. Before running a job, you can view the SQL that Data Integrator generates and adjust your design to maximize the SQL that is pushed down to improve performance. This section discusses: Push down logic Push down example Viewing SQL

Push Down Logic

By pushing down operations to the source or target database, Data Integrator reduces the number of rows and operations that the engine must retrieve and process, which improves performance. When determining which operations to push to the database, Data Integrator examines the database and its environment. Operations that Data Integrator pushes to the database include: Aggregations Aggregate functions, typically used with a Group by statement, always produce a data set smaller or the same size as the original data set. Distinct rows When you select Distinct rows from the Select tab in the query editor, Data Integrator will only output unique rows. Filtering Filtering may produce a data set smaller than or equal to the original data set. Joins Joins typically produce a data set smaller or similar in size to the original tables. Projection Projection normally produces a smaller data set since it only returns columns referenced by a data flow. Functions Most Data Integrator functions that have equivalents in the underlying database are appropriately translated. Data Integrator cannot push some transform operations to the database. For example: Expressions that include Data Integrator functions that do not have database correspondents Load operations that contain triggers Data Integrator can only push operations supported by the DBMS down to that DBMS. Therefore, for best performance, try not to intersperse Data Integrator transforms among operations that can be pushed down to the database. For example, any data flow step containing a Data Integrator custom function or a built-in transform cannot be pushed to the source database.

NOTE: Database-specific functions can only be used in situations where they will be pushed down to the database for execution.

Push Down Example

When determining how to push operations to the database, Data Integrator first collapses all the transforms into the minimum set of transformations expressed in terms of the source table columns. Next, Data Integrator pushes all possible operations on tables of the same database down to that DBMS. For example, the following data flow extracts rows from a single source table. The first query selects only the rows in the source where column A contains a value greater than 100. The second query refines the extraction further, reducing the number of columns returned and further reducing the qualifying rows. Data Integrator collapses the two queries into a single command for the DBMS to execute: SELECT A, MAX(B), C FROM source WHERE A > 100 AND B = C GROUP BY A, C SELECT A, MAX(B), C FROM source WHERE B = C GROUP BY A, C SELECT * FROM source WHERE A>100 All the operations in this data flow could be passed to the DBMS. However, if the data flow contains operations that cannot be passed to the DBMS, Data Integrator optimizes the transformation differently. For example, if Query1 called func(A) > 100, where func is a Data Integrator custom function, then Data Integrator would generate two commands. The source DBMS executes the first command. The first query becomes: SELECT A, B, C FROM source WHERE B = C Data Integrator executes the second command. The second query becomes: SELECT A, MAX(B), C FROM Query1 WHERE func(A) > 100 GROUP BY A, C Viewing SQL Before running a job, you can view the SQL code Data Integrator generates for table sources in data flows. By examining the SQL code, you can verify that Data Integrator generates the commands you expect. If necessary, you can alter your design to improve the data flow.

The pushdown_sql function overview from the Data Integrator technical guide.