DataWareHousing - ETL Project Life Cycle

Posted by Suresh at 1:! "# Labels: Data Warehouse$ ETL$ Life Cycle
The follo%ing are the different &hases in'ol'ed in a ETL &roject de'elo&(ent life cycle)
1* +e,uire(ent -athering
a* .usiness +e,uire(ent Collection / .+D *
b* Syste( +e,uire(ent Collection / S+D *
0* Design Phase
a* High Le'el Design Docu(ent / H+D *
b* Lo% le'el Design Docu(ent / LLD *
1* De'elo&(ent Phase
a* #a&&ing Design
b* Code +e'ie%
c* Peer +e'ie%
2* Testing
a* 3nit Testing
b* Syste( 4ntegration Testing)
c* 3Ser "cce&tance Testing / 3"T *
5* Pre - Production
6* Production / -o-Li'e *
+e,uire(ent -athering:
---------------------------------------------
.usiness +e,uire(ent Collection / .+D *:
-7 The business re,uire(ent gathering start by business "nalyst$ onsite technical lead and client
business users)
-7 4n this &hase$a .usiness "nalyst &re&ares .usiness +e,uire(ent Docu(ent / .+D * /or*
.usiness
+e,uire(ent S&ecifications / .+S *
-7 .+ collection ta8es &lace at client location)
-7 The o9& fro( .+ "nalysis are
-7 .+S .usiness +e,uire(ent S&ecifications
-7 S+S Syste( +e,uire(ent S&ecifications
Syste( +e,uire(ent Collection / S+D *:
Senior technical &eo&le /or* ETL architect %ill &re&are the S+S %hich contains s9% and h9%
details
The S+S %ill includes
a* :9S to be used / %indo%s or uni; *
b* +D.#S re,uired to build database / oracle$ Teradata etc *
c* ETL tools re,uired / 4nfor(atica$Datastage *
d* :L"P tools re,uired / Cognos $.: *
The S+S is also called as Technical +e,uire(ent S&ecifications / T+S *

Design Phase:
---------------------------------------
-7 The o9& fro( design and &lanning &hase is
a* HLD / High Le'el Design * Docu(ent
b*LLD / Lo% Le'el Design * Docu(ent
HLD / High Le'el Design * Docu(ent : -
"n ETL "rchitect and DWH "rchitect &artici&ate in designing a solution to build a DWH)
"n HLD docu(ent is &re&ared based on .usiness +e,uire(ent)
LLD / Lo% Le'el Design * Docu(ent : -
.ased on HLD$a senior ETL de'elo&er &re&are Lo% Le'el Design Docu(ent
The LLD contains (ore technical details of an ETL Syste()
"n LLD contains data flo% diagra( / D<D *$ details of source and targets of each (a&&ing)
"n LLD also contains infor(ation about full and incre(ental load)
"fter LLD then De'elo&(ent Phase %ill start
De'elo&(ent Phase :
--------------------------------------------------
#a&&ing:
-7 .ased an LLD$ the ETL tea( %ill create (a&&ing / ETL Code *
-7 "fter designing the (a&&ings$ the code / #a&&ings * %ill be re'ie%ed by de'elo&ers)
Code +e'ie% :-
-7 Code +e'ie% %ill be done by de'elo&er)
-7 4n code re'ie%$the de'elo&er %ill re'ie% the code and the logic but not the data)
-7 The follo%ing acti'ities ta8es &lace in code re'ie%
-7 =ou ha'e to chec8 the na(ing standards of transfor(ation$(a&&ings of data etc)
-7 Source and target (a&&ing / Placed the correct logic or not in (a&&ing *
Peer +e'ie% :-
-7 The code %ill re'ie%ed by your tea( (e(ber / third &arty de'elo&er *
Testing:
---------------
The follo%ing 'arious ty&es testing carried out in testing en'iron(ent)
1* 3nit Testing
0* De'elo&(ent 4ntegration Testing
1* Syste( 4ntegration Testing
2* 3ser "cce&tance Testing
3nit Testing :-
-7 " unit test for the DWH is a %hite .o; testing$4t should chec8 the ETL &rocedure and
#a&&ings)
-7 The follo%ing are the test cases can be e;ecuted by an ETL de'elo&er)
1* >erify data loss
0* ?o)of records in the source and target
1* Dataload94nsert
2* Dataload93&date
5* 4ncre(ental load
6* Data accuracy
@* 'erify ?a(ing standards)
!* >erify colu(n #a&&ing
-7 The 3nit Test %ill be carried by ETL de'elo&er in de'elo&(ent &hase)
-7 ETL de'elo&er has to do the data 'alidations also in this &hase)
De'elo&(ent 4ntegration Testing -
-7 +un all the (a&&ings in the se,uence order)
-7 <irst +un the source to stage (a&&ings)
-7 Then run the (a&&ings related to di(ensions and facts)
Syste( 4ntegration Testing :-
-7 "fter de'elo&(ent &hase$%e ha'e to (o'e our code to A" en'iron(ent)
-7 4n this en'iron(ent$%e are gi'ing read-only &er(ission to testing &eo&le)
-7 They %ill test all the %or8flo%s)
-7 "nd they %ill test our code according to their standards)
3ser "cce&tance Testing / 3"T * :-
-7 This test is carried out in the &resence of client side technical users to 'erify the data
(igration fro( source to destination)

Production En'iron(ent :-
---------------------------------
-7 #igrate the code into the -o-Li'e en'iron(ent fro( test en'iron(ent / A" En'iron(ent *)