ETL Detailed Design Document

ETL Detailed Design Template V1.0

Document Details
Project Name
Current Version
List of Contributors
Customer Contact Information

Prepared by/Date

Reviewed by/Date

Approved by/Date

Revision History
Version

Date of
Revision

Revised
By

Description of
Change

Reason for
Change

Affected
Sections

Approved
By

Affected Groups

DRAFT

Confidential

Page 2 of 15

ETL Detailed Design Template V1.0

Table of Contents
1

INTRODUCTION

1.1

DOCUMENT PURPOSE AND SCOPE.................................................................................

1.2

PROJECT BACKGROUND................................................................................................

1.3

PROJECT SCOPE..........................................................................................................

1.4

DATA SOURCES AND CONTACT DETAILS.........................................................................

1.5

REFERENCES................................................................................................................

TOOLS

LOAD STRATEGY

3.1

OVERVIEW....................................................................................................................
3.1.1

Initial Load........................................................................................................

3.1.2

Incremental Load..............................................................................................

DETAILED ETL DESIGN

4.1

<MODULE NAME/SUBJECT AREA>.................................................................................

4.1.1

Source Definition..............................................................................................

4.1.2

Target Definition................................................................................................

4.1.3

Transformation and Load..................................................................................

4.2

PROCESSING LOGIC......................................................................................................
4.2.1

Process Flow....................................................................................................

4.2.2

Error Handling...................................................................................................

4.2.3

Reconciliation Logic..........................................................................................

4.2.4

Dependencies...................................................................................................

RESTART AND RECOVERY

10

GENERAL PURPOSE ROUTINES

11

6.1

DRAFT

<ROUTINE>................................................................................................................
6.1.1

Input Parameters............................................................................................

6.1.2

Output Parameters..........................................................................................
Confidential

Page 3 of 15

ETL Detailed Design Template V1.0

6.1.3

Frequency.......................................................................................................

6.1.4

Error Handling.................................................................................................

6.1.5

Notification......................................................................................................

6.1.6

Dependencies.................................................................................................

SCHEDULER
7.1

12

SCHEDULING INTERFACE.............................................................................................

DATA RETENTION AND ARCHIVING

13

TRACEABILITY MATRIX

14

10

ACRONYMS AND GLOSSARY

15

DRAFT

Confidential

Page 4 of 15

ETL Detailed Design Template V1.0

1 Introduction
Brief introduction about the project

1.1 Document Purpose and Scope

Brief the objective of the project

1.2 Project Background

1.3 Project Scope

Brief the scope of the project

1.4 Data Sources and Contact Details

Data Source Name

Contact Details

1.5 References
Document name

DRAFT

Author

Version / Date

Confidential

Location of document

Page 5 of 15

ETL Detailed Design Template V1.0

2 Tools
List the tools used, if any

DRAFT

Confidential

Page 6 of 15

ETL Detailed Design Template V1.0

3 Load Strategy
3.1 Overview
Detail the load strategy.

3.1.1

Initial Load
Describe the first time load.

3.1.2

Incremental Load
Describe the incremental loading process.

3.1.2.1

Full Refresh
Describe the incremental loading which fully refreshes the data in the tables after the first time load

3.1.2.2

Delta
Describe delta incremental loading

DRAFT

Confidential

Page 7 of 15

ETL Detailed Design Template V1.0

4 Detailed ETL Design

This section should describe in detail the extraction, cleansing, transformation and load strategy for all the modules
and the programs within them.

4.1 <Module Name/Subject Area>

This section should describe in detail the extraction, transformation, cleansing and load strategy for the subject area
/module under consideration with the data flow diagram.

4.1.1

Source Definition

Table/File
Name

Column/Field
Name

Column/Field
Description

Data Type

Length

Numeric/AlphaNumeric

4.1.2

Target Definition

Table/File
Name

Column/Field
Name

Column/Field
Description

Data Type

Length

Numeric/AlphaNumeric

4.1.3

Transformation and Load

This section should describe in detail the transformation and load process - to load the target tables/files(Staging
Area or Data warehouse database) using all the source tables/files .

Target
Target
Table/ File Column/
Name
Field
Name

DRAFT

Target
Target
Master
Source
Source
Data Transformation
Column/ Column/ Field Table/ File Table / File Column/ Field
Field
Data
Lookup
Name
Name
Default
Validation information
value
checks
(null/length,
data type)

Confidential

Page 8 of 15

ETL Detailed Design Template V1.0

4.2 Processing Logic

4.2.1

Process Flow

Process flow will contain the algorithm or a similar step by step representation of the processing done by the routine.

4.2.2

Error Handling
This section should detail the error handling for the program under consideration.

4.2.3

Reconciliation Logic

This section explains the logic used to keep track of the number of records fed, number of records processed, number of valid
records, number of records corrected and number of records rejected and how they are reconciled.

4.2.4

Dependencies
This section should detail the dependencies of the program under consideration with other programs or General
purpose routines or any other external system.

DRAFT

Confidential

Page 9 of 15

ETL Detailed Design Template V1.0

5 Restart and Recovery

Detail the restart and recovery steps

DRAFT

Confidential

Page 10 of 15

ETL Detailed Design Template V1.0

6 General Purpose Routines

Detail each routine that will be used in general (ex: date conversion, some specific validations across programs etc)
or for data extraction or for data loading.

6.1 <Routine>
Detail the scope of the routine and its funtionality.

6.1.1

Input Parameters
Detail the input parameters used in the routine.

6.1.2

Output Parameters
Detail the outputs of the routine.

6.1.3

Frequency
Detail the frequency and time when this routine will be executed if independent.

6.1.4

Error Handling
This section should detail the error handling.

6.1.5

Notification
This section should brief the notification procedure.

6.1.6

Dependencies
This section should brief the dependencies on programs, sub-programs or any other external system.

DRAFT

Confidential

Page 11 of 15

ETL Detailed Design Template V1.0

7 Scheduler
7.1 Scheduling Interface
Details of schedullng tool to be used and the schedules for the programs.

DRAFT

Confidential

Page 12 of 15

ETL Detailed Design Template V1.0

8 Data Retention and Archiving

This section gives details of data retention and archiving.

DRAFT

Confidential

Page 13 of 15

ETL Detailed Design Template V1.0

Traceability Matrix
This Section lists the traceability information to trace the requirements from Requirements Analysis document to
Design document. The locations of design specifications should be mapped to the corresponding sections of the
requirements in the Requirements Analysis document.

Document reference Id & Description: (Doc Id from which this document is derived)

Serial
No.

DRAFT

Reference document
Requirement/Feature (Section ID/Name)

Confidential

Design document
(Section ID/Name)

Page 14 of 15

ETL Detailed Design Template V1.0

10 Acronyms and Glossary

This section describes the acronyms of various technical terms used in the document.

DRAFT

Confidential

Page 15 of 15