Best Practices for Designing SQL*Loader Mappings

This section includes the following topics:

Using Conventional Loading to Ensure Referential Integrity in SQL*Loader Mappings Using Direct Path Loading to Ensure Referential Integrity in SQL*Loader Mappings

Using Conventional Loading to Ensure Referential Integrity in SQL*Loader Mappings

If you are extracting data from a multiple-record-type file with a master-detail structure and mapping to tables, add a Mapping Sequence operator to the mapping to retain the relationship between the master and detail records through a surrogate primary key or foreign key relationship. A master-detail file structure is one where a master record is followed by its detail records. In Example 22-1, records beginning with "E" are master records with Employee information and records beginning with "P" are detail records with Payroll information for the corresponding employee.

Example 22-1 A Multiple-Record-Type Flat File with a Master-Detail Structure

E P P E P E P P

003715 4 01152000 02152000 003941 2 03152000 001939 2 01152000 02152000

153 09061987 014000000 "IRENE HIRSH" 1 08500 01162000 00101 000500000 000700000 02162000 00102 000300000 000800000 165 03111959 016700000 "ANNE FAHEY" 1 09900 03162000 00107 000300000 001000000 265 09281988 021300000 "EMILY WELLMET" 1 07700 01162000 00108 000300000 001000000 02162000 00109 000300000 001000000

In Example 22-1, the relationship between the master and detail records is inherent only in the physical record order: payroll records correspond to the employee record they follow. However, if this is the only means of relating detail records to their masters, this relationship is lost when Warehouse Builder loads each record into its target table.

Maintaining Relationships Between Master and Detail Records

You can maintain the relationship between master and detail records if both types of records share a common field. If Example 22-1 contains a field Employee ID in both

Employee and Payroll records, you can use it as the primary key for the Employee table and as the foreign key in the Payroll table, thus associating Payroll records to the correct Employee record. However, if your file does not have a common field that can be used to join master and detail records, you must add a sequence column to both the master and detail targets (see Table 22-5 and Table 22-6) to maintain the relationship between the master and detail records. Use the Mapping Sequence operator to generate this additional value. Table 22-5 represents the target table containing the master records from the file in Example 22-1. The target table for the master records in this case contains employee information. Columns E1-E10 contain data extracted from the flat file. Column E11 is the additional column added to store the master sequence number. Notice that the number increments by one for each employee.

Table 22-5 Target Table Containing Master Records

E1 E2 E3 E4 E5 E6 E7 E8 E9 E10 E11

E E E

003715 4 003941 2 001939 2

153 09061987 014000000 "IRENE HIRSH" 1 165 03111959 016700000 "ANNE FAHEY" 1 265 09281988 021300000 "EMILY WELSH" 1

08500 1 09900 2 07700 3

Table 22-6 represents the target table containing the detail records from the file in Example 22-1. The target table for the detail records in this case contains payroll information, with one or more payroll records for each employee. Columns P1-P6 contain data extracted from the flat file. Column P7 is the additional column added to store the detail sequence number. Notice that the number for each payroll record matches the corresponding employee record in Table 22-5.

Table 22-6 Target Table Containing Detail Records

P1 P2 P3 P4 P5 P6 P7

P P P
P P

01152000 02152000 03152000 01152000 02152000

01162000 02162000 03162000 01162000 02162000

00101 00102 00107 00108 00109

000500000 000300000 000300000 000300000 000300000

000700000 000800000 001000000 001000000 001000000

1 1 2
3 3

Extracting and Loading Master-Detail Records

This section contains instructions on creating a mapping that extracts records from a master-detail flat file and loads those records into two different tables. One target table stores master records and the other target table stores detail records from the flat file. The Mapping Sequence is used to maintain the master-detail relationship between the two tables. Note: These instructions are for conventional path loading. For instructions on using direct path loading for master-detail records, see "Using Direct Path Loading to Ensure Referential Integrity in SQL*Loader Mappings". This procedure outlines general steps for building such a mapping. Additional detailed instructions are available at:

Using the Import Metadata Wizard Flat File Operator Adding Operators that Bind to Workspace Objects Sequence Operator "Configuring Mappings Reference"

To extract from a master-detail flat file and maintain master-detail relationships, use the following steps: 1. Import and sample the flat file source that consists of master and detail records. When naming the record types as you sample the file, assign descriptive names to the master and detail records. This makes it easier to identify those records in the future. Figure 22-9 shows the Flat File Sample Wizard for a multiple-record-type flat file containing department and employee information. The master record type (for employee records) is called EmployeeMaster, while the detail record type (for payroll information) is called PayrollDetail.

Figure 22-9 Naming Flat File Master and Detail Record Types

Description of "Figure 22-9 Naming Flat File Master and Detail Record Types " 2. Drop a Flat File operator onto the Mapping Editor canvas and specify the master-detail file from which you want to extract data. 3. Drop a Sequence operator onto the mapping canvas. 4. Drop a Table operator for the master records onto the mapping canvas. You can either select an existing workspace table that you created earlier or create a new unbound table operator with no attributes. You can then map or copy all required fields from the master record of the file operator to the master table operator (creating columns) and perform an outbound reconciliation to define the table later. The table must contain all the columns required for the master fields you want to load plus an additional numeric column for loading sequence values.

5. Drop a Table operator for the detail records onto the mapping canvas. You can either select an existing workspace table that you created earlier or create a new unbound table operator with no attributes. You can then map or copy all required fields from the master record of the file operator to the master table operator (creating columns) and perform an outbound synchronize to define the table later. The table must contain all the columns required for the detail fields you want to load plus an additional numeric column for loading sequence values. 6. Map all of the necessary flat file master fields to the master table and detail fields to the detail table. Figure 22-10 displays the mapping of the fields. 7. Map the Sequence NEXTVAL attribute to the additional sequence column in the master table. Figure 22-10 displays the mapping from the NEXTVAL attribute of the Sequence operator to the master table. 8. Map the Sequence CURRVAL attribute to the additional sequence column in the detail table. Figure 22-10 shows a completed mapping with the flat file master fields mapped to the master target table, the detail fields mapped to the detail target table, and the NEXTVAL and CURRVAL attributes from the Mapping Sequence mapped to the master and detail target tables, respectively.

Figure 22-10 Completed Mapping from Master-Detail Flat File to Two Target Tables

Description of "Figure 22-10 Completed Mapping from Master-Detail Flat File to Two Target Tables" 9. Configure the mapping that loads the source data into the target tables with the following parameters: Direct Mode: Not selected Errors Allowed: 0 Row: 1 Trailing Nullcols: True (for all tables)

Error Handling Suggestions

This section contains error handling recommendations for files with varying numbers of errors.

If your data file almost never contains errors: 1. Create a mapping with a Sequence operator (see "Sequence Operator"). 2. Configure a mapping with the following parameters: Direct Mode= Not selected ROW=1 ERROR ALLOWED = 0 3. Generate the code and run an SQL*Loader script. If the data file has errors, then the loading stops when the first error occurs. 4. Fix the data file and run the control file again with the following configuration values: CONTINUE_LOAD=TRUE SKIP=number of records already loaded If your data file is likely to contain a moderate number of errors: 1. Create a primary key (PK) for the master record based on the seq_nextval column. 2. Create a foreign key (FK) for the detail record based on the seq_currval column which references the master table PK. In this case, master records with errors will be rejected with all their detail records. You can recover these records by following these steps. 3. Delete all failed detail records that have no master records. 4. Fix the errors in the bad file and reload only those records. 5. If there are very few errors, you may choose to load the remaining records and manually update the table with correct sequence numbers. 6. In the log file, you can identify records that failed with errors because those errors violate the integrity constraint. The following is an example of a log file record with errors:

7. Record 9: Rejected - Error on table "MASTER_T", column "C3". 8. ORA-01722: invalid number 9. Record 10: Rejected - Error on table "DETAIL1_T".

10. ORA-02291: integrity constraint (SCOTT.FK_SEQ) violated - parent key not found 11. Record 11: Rejected - Error on table "DETAIL1_T". 12. ORA-02291: integrity constraint (SCOTT.FK_SEQ) violated - parent key not found 13. Record 21: Rejected - Error on table "DETAIL2_T". 14. ORA-02291: invalid number
If your data file always contains many errors: 1. Load all records without using the Sequence operator. Load the records into independent tables. You can load the data in Direct Mode, with the following parameters that increase loading speed: ROW>1 ERRORS ALLOWED=MAX 2. Correct all rejected records. 3. Reload the file again with a Sequence operator (see "Sequence Operator").

Subsequent Operations
After the initial loading of the master and detail tables, you can use the loaded sequence values to further transform, update, or merge master table data with detail table data. For example, if your master records have a column that acts as a unique identifier, such as an Employee ID, and you want to use it as the key to join master and detail rows (instead of the sequence field you added for that purpose), you can update the detail tables to use this unique column. You can then drop the sequence column you created for the purpose of the initial load. Operators such as the Aggregator, Filter, or Match and Merge operator can help you with these subsequent transformations.

Using Direct Path Loading to Ensure Referential Integrity in SQL*Loader Mappings

If you are using a master-detail flat file where the master record has a unique field (or if the concatenation of several fields can result in a unique identifier), you can use Direct Path Load as an option for faster loading.

For direct path loading, the record number (RECNUM) of each record is stored in the master and detail tables. A post-load procedure uses the RECNUM to update each detail row with the unique identifier of the corresponding master row. This procedure outlines general steps for building such a mapping. Additional detailed instructions are available:

For additional information on Metadata Wizard". For additional information on Operator". For additional information on Bind to Workspace Objects". For additional information on Generator Operator". For additional information on Operator". For additional information on Reference".

importing flat file sources, see "Using the Import using the Flat File as a source, see "Flat File using Table operators, see "Adding Operators that using the Data Generator operator, see "Data using the Constant operator, see "Constant configuring mappings, see "Configuring Mappings

To extract from a master-detail flat file using direct path load to maintain master-detail relationships: 1. Import and sample a flat file source that consists of master and detail records. When naming the record types as you sample the file, assign descriptive names to the master and detail records, as shown in Figure 22-9. This will make it easier to identify those records in the future. 2. Create a mapping that you will use to load data from the flat file source. 3. Drop a Flat File operator onto the mapping canvas and specify the master-detail file from which you want to extract data. 4. Drop a Data Generator and a Constant operator onto the mapping canvas. 5. Drop a Table operator for the master records onto the mapping canvas. You can either select an existing workspace table that you created earlier, or create a new unbound table operator with no attributes and perform an outbound synchronize to define the table later. The table must contain all the columns required for the master fields you plan to load plus an additional numeric column for loading the RECNUM value. 6. Drop a Table operator for the detail records onto the mapping canvas.

You can either select an existing workspace table that you created earlier, or create a new unbound table operator with no attributes and perform an outbound synchronize to define the table later. The table must contain all the columns required for the detail fields you plan to load plus an additional numeric column for loading a RECNUM value, and a column that will be updated with the unique identifier of the corresponding master table row. 7. Map all of the necessary flat file master fields to the master table and detail fields to the detail table, as shown in Figure 22-12. 8. Map the Data Generator operator's RECNUM attribute to the RECNUM columns in the master and detail tables, as shown in Figure 22-12. 9. Add a constant attribute in the Constant operator. If the master row unique identifier column is of a CHAR data type, make the constant attribute a CHAR type with the expression '*'. If the master row unique identifier column is a number, make the constant attribute a NUMBER with the expression '0'. Figure 22-11 shows the expression property of the constant attribute set to '0'. This constant marks all data rows as "just loaded".

Figure 22-11 Constant Operator Properties

Description of "Figure 22-11 Constant Operator Properties" 10. Map the constant attribute from the Constant operator to the detail table column that will later store the unique identifier for the corresponding master table record. Figure 22-12 shows a completed mapping with the flat file's master fields mapped to the master target table, the detail fields mapped to the detail target

table, the RECNUM attributes from the Data Generator operator mapped to the master and detail target tables, respectively, and the constant attribute mapped to the detail target table.

Figure 22-12 Completed Mapping from Master-Detail Flat File with a Direct Path Load

Description of "Figure 22-12 Completed Mapping from Master-Detail Flat File with a Direct Path Load" 11. Configure the mapping with the following parameters: Direct Mode: True Errors Allowed: 0

Trailing Nullcols: True (for each table) 12. After you validate the mapping and generate the SQL*Loader script, create a post-update PL/SQL procedure and add it to the Warehouse Builder library. 13. Run the SQL*Loader script. 14. Execute an UPDATE SQL statement by running a PL/SQL post-update procedure or manually executing a script. The following is an example of the generated SQL*Loader control file script:

OPTIONS ( DIRECT=TRUE,PARALLEL=FALSE, ERRORS=0, BINDSIZE=50000, ROWS=200, READSIZE=65536) LOAD DATA CHARACTERSET WE8MSWIN1252 INFILE 'g:\FFAS\DMR2.dat' READBUFFERS 4 INTO TABLE "MATER_TABLE" APPEND REENABLE DISABLED_CONSTRAINTS WHEN "REC_TYPE"='P' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' TRAILING NULLCOLS ( "REC_TYPE" POSITION (1) CHAR , "EMP_ID" CHAR , "ENAME" CHAR , "REC_NUM" RECNUM ) INTO TABLE "DETAIL_TABLE" APPEND REENABLE DISABLED_CONSTRAINTS WHEN "REC_TYPE"='E' FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"' TRAILING NULLCOLS ( "REC_TYPE" POSITION (1) CHAR ,

"C1" CHAR , "C2" CHAR , "C3" CHAR , "EMP_ID" CONSTANT '*', "REC_NUM" RECNUM
The following is an example of the post-update PL/SQL procedure:

create or replace procedure wb_md_post_update( master_table varchar2 ,master_recnum_column varchar2 ,master_unique_column varchar2 ,detail_table varchar2 ,detail_recnum_column varchar2 ,detail_masterunique_column varchar2 ,detail_just_load_condition varchar2) IS v_SqlStmt VARCHAR2(1000); BEGIN v_SqlStmt := 'UPDATE '||detail_table||' l '|| ' SET l.'||detail_masterunique_column||' = (select i.'||master_unique_column|| ' from '||master_table||' i '|| ' WHERE i.'||master_recnum_column||' IN '|| ' (select max(ii.'||master_recnum_column||') '|| ' from '||master_table||' ii '|| ' WHERE ii.'||master_recnum_column||' < l.'||detail_recnum_column||') '|| ' ) '|| ' WHERE l.'||detail_masterunique_column||' = '||''''||detail_just_load_condition||''''; dbms_output.put_line(v_sqlStmt); EXECUTE IMMEDIATE v_SqlStmt; END;