Push Down Optimisation

6/5/2015
krishnareddyoracleapps.blogspot.in/search/label/INFORMATICA
PUSHDOWNOPTIMISATION
Youcanpushtransformationlogictothesourceortargetdatabaseusingpushdownoptimization.Whenyourunasessionconfiguredforpushdown
optimization,theIntegrationServicetranslatesthetransformationlogicintoSQLqueriesandsendstheSQLqueriestothedatabase.Thesourceor
targetdatabaseexecutestheSQLqueriestoprocessthetransformations.
Theamountoftransformationlogicyoucanpushtothedatabasedependsonthedatabase,transformationlogic,andmappingandsession
configuration.TheIntegrationServiceprocessesalltransformationlogicthatitcannotpushtoadatabase.
UsethePushdownOptimizationViewertopreviewtheSQLstatementsandmappinglogicthattheIntegrationServicecanpushtothesourceortarget
database.YoucanalsousethePushdownOptimizationViewertoviewthemessagesrelatedtopushdownoptimization.
Thefollowingfigureshowsamappingcontainingtransformationlogicthatcanbepushedtothesourcedatabase:
ThismappingcontainsanExpressiontransformationthatcreatesanitemIDbasedonthestorenumber5419andtheitemIDfromthesource.Topush
thetransformationlogictothedatabase,theIntegrationServicegeneratesthefollowingSQLstatement:
INSERTINTOT_ITEMS(ITEM_ID,ITEM_NAME,ITEM_DESC)SELECTCAST((CASEWHEN5419ISNULLTHEN''ELSE5419END)+'_'
+(CASEWHENITEMS.ITEM_IDISNULLTHEN''ELSEITEMS.ITEM_IDEND)ASINTEGER),ITEMS.ITEM_NAME,ITEMS.ITEM_DESC
FROMITEMS2ITEMS
TheIntegrationServicegeneratesanINSERTSELECTstatementtoretrievetheID,name,anddescriptionvaluesfromthesourcetable,createnew
itemIDs,andinsertthevaluesintotheITEM_ID,ITEM_NAME,andITEM_DESCcolumnsinthetargettable.Itconcatenatesthestorenumber
5419,anunderscore,andtheoriginalITEMIDtogetthenewitemID.
PushdownOptimizationTypes
Youcanconfigurethefollowingtypesofpushdownoptimization:
Sourcesidepushdownoptimization.TheIntegrationServicepushesasmuchtransformationlogicaspossibletothesourcedatabase.
Targetsidepushdownoptimization.TheIntegrationServicepushesasmuchtransformationlogicaspossibletothetargetdatabase.
Fullpushdownoptimization.TheIntegrationServiceattemptstopushalltransformationlogictothetargetdatabase.IftheIntegrationService
cannotpushalltransformationlogictothedatabase,itperformsbothsourcesideandtargetsidepushdownoptimization.
RunningSourceSidePushdownOptimizationSessions
Whenyourunasessionconfiguredforsourcesidepushdownoptimization,theIntegrationServiceanalyzesthemappingfromthesourcetothetarget
http://krishnareddyoracleapps.blogspot.in/search/label/INFORMATICA
1/51
6/5/2015
oruntilitreachesadownstreamtransformationitcannotpushtothesourcedatabase.
TheIntegrationServicegeneratesandexecutesaSELECTstatementbasedonthetransformationlogicforeachtransformationitcanpushtothe
database.Then,itreadstheresultsofthisSQLqueryandprocessestheremainingtransformations.
RunningTargetSidePushdownOptimizationSessions
Whenyourunasessionconfiguredfortargetsidepushdownoptimization,theIntegrationServiceanalyzesthemappingfromthetargettothesource
oruntilitreachesanupstreamtransformationitcannotpushtothetargetdatabase.ItgeneratesanINSERT,DELETE,orUPDATEstatementbased
onthetransformationlogicforeachtransformationitcanpushtothetargetdatabase.TheIntegrationServiceprocessesthetransformationlogicupto
thepointthatitcanpushthetransformationlogictothedatabase.Then,itexecutesthegeneratedSQLontheTargetdatabase.
RunningFullPushdownOptimizationSessions
Tousefullpushdownoptimization,thesourceandtargetdatabasesmustbeinthesamerelationaldatabasemanagementsystem.Whenyouruna
sessionconfiguredforfullpushdownoptimization,theIntegrationServiceanalyzesthemappingfromthesourcetothetargetoruntilitreachesa
downstreamtransformationitcannotpushtothetargetdatabase.ItgeneratesandexecutesSQLstatementsagainstthesourceortargetbasedonthe
transformationlogicitcanpushtothedatabase.
Whenyourunasessionwithlargequantitiesofdataandfullpushdownoptimization,thedatabaseservermustrunalongtransaction.Considerthe
followingdatabaseperformanceissueswhenyougeneratealongtransaction:
Alongtransactionusesmoredatabaseresources.
Alongtransactionlocksthedatabaseforlongerperiodsoftime.Thisreducesdatabaseconcurrencyandincreasesthelikelihoodofdeadlock.
Alongtransactionincreasesthelikelihoodofanunexpectedevent.Tominimizedatabaseperformanceissuesforlongtransactions,consider
usingsourcesideortargetsidepushdownoptimization.
RulesandGuidelinesforFunctionsinPushdownOptimization
Usethefollowingrulesandguidelineswhenpushingfunctionstoadatabase:
IfyouuseADD_TO_DATEintransformationlogictochangedays,hours,minutes,orseconds,youcannotpushthefunctiontoaTeradata
database.
WhenyoupushLAST_DAY()toOracle,Oraclereturnsthedateuptothesecond.Iftheinputdatecontainssubseconds,Oracletrimsthedate
tothesecond.
WhenyoupushLTRIM,RTRIM,orSOUNDEXtoadatabase,thedatabasetreatstheargument('')asNULL,buttheIntegrationServicetreats
theargument('')asspaces.
AnIBMDB2databaseandtheIntegrationServiceproducedifferentresultsforSTDDEVandVARIANCE.IBMDB2usesadifferent
algorithmthanotherdatabasestocalculateSTDDEVandVARIANCE.
WhenyoupushSYSDATEorSYSTIMESTAMPtothedatabase,thedatabaseserverreturnsthetimestampinthetimezoneofthedatabase
server,nottheIntegrationService.
IfyoupushSYSTIMESTAMPtoanIBMDB2oraSybasedatabase,andyouspecifytheformatforSYSTIMESTAMP,thedatabaseignores
theformatandreturnsthecompletetimestamp.
YoucanpushSYSTIMESTAMP(SS)toaNetezzadatabase,butnotSYSTIMESTAMP(MS)orSYSTIMESTAMP(US).
2/51
6/5/2015
WhenyoupushTO_CHAR(DATE)orTO_DATE()toNetezza,dateswithsubsecondprecisionmustbeintheYYYYMMDDHH24:MI:
SS.USformat.Iftheformatisdifferent,theIntegrationServicedoesnotpushthefunctiontoNetezza.
PERFORMANCETUNINGOFLOOKUPTRANSFORMATIONS
Lookuptransformationsareusedtolookupasetofvaluesinanothertable.Lookupsslowsdowntheperformance.
1.Toimproveperformance,cachethelookuptables.Informaticacancacheallthelookupandreferencetablesthismakesoperationsrunveryfast.
(Meaningofcacheisgiveninpoint2ofthissectionandtheprocedurefordeterminingtheoptimumcachesizeisgivenattheendofthisdocument.)
2.Evenaftercaching,theperformancecanbefurtherimprovedbyminimizingthesizeofthelookupcache.Reducethenumberofcachedrowsby
usingasqloverridewitharestriction.
Cache:CachestoresdatainmemorysothatInformaticadoesnothavetoreadthetableeachtimeitisreferenced.Thisreducesthetimetakenbythe
processtoalargeextent.CacheisautomaticallygeneratedbyInformaticadependingonthemarkedlookupportsorbyauserdefinedsqlquery.
Exampleforcachingbyauserdefinedquery:
Supposeweneedtolookuprecordswhereemployee_id=eno.
employee_idisfromthelookuptable,EMPLOYEE_TABLEandenoisthe
inputthatcomesfromthefromthesourcetable,SUPPORT_TABLE.
WeputthefollowingsqlqueryoverrideinLookupTransform
selectemployee_idfromEMPLOYEE_TABLE
Ifthereare50,000employee_id,thensizeofthelookupcachewillbe50,000.
Insteadoftheabovequery,weputthefollowing:
selectempemployee_idfromEMPLOYEE_TABLEe,SUPPORT_TABLEs
wheree.employee_id=s.eno
Ifthereare1000eno,thenthesizeofthelookupcachewillbeonly1000.Butheretheperformancegainwillhappenonlyifthenumberofrecordsin
SUPPORT_TABLEisnothuge.Ourconcernistomakethesizeofthecacheaslessaspossible.
3.Inlookuptables,deleteallunusedcolumnsandkeeponlythefieldsthatareusedinthemapping.
4.Ifpossible,replacelookupsbyjoinertransformationorsinglesourcequalifier.Joinertransformationtakesmoretimethansourcequalifier
transformation.
5.Iflookuptransformationspecifiesseveralconditions,thenplaceconditionsthatuseequalityoperator=firstintheconditionsthatappearinthe
conditionstab.
6.Inthesqloverridequeryofthelookuptable,therewillbeanORDERBYclause.Removeitifnotneededorputfewercolumnnamesinthe
ORDERBYlist.
7.Donotusecachinginthefollowingcases:
Sourceissmallandlookuptableislarge.
Iflookupisdoneontheprimarykeyofthelookuptable.
8.Cachethelookuptablecolumnsdefinitelyinthefollowingcase:
Iflookuptableissmallandsourceislarge.
9.Iflookupdataisstatic,usepersistentcache.Persistentcacheshelptosaveandreusecachefiles.Ifseveralsessionsinthesamejobusethesame
3/51
6/5/2015
lookuptable,thenusingpersistentcachewillhelpthesessionstoreusecachefiles.Incaseofstaticlookups,cachefileswillbebuiltfrommemory
cacheinsteadoffromthedatabase,whichwillimprovetheperformance.
10.Ifsourceishugeandlookuptableisalsohuge,thenalsousepersistentcache.
11.Iftargettableisthelookuptable,thenusedynamiccache.TheInformaticaserverupdatesthelookupcacheasitpassesrowstothetarget.
12.Useonlythelookupsyouwantinthemapping.Toomanylookupsinsideamappingwillslowdownthesession.
13.Iflookuptablehasalotofdata,thenitwilltaketoolongtocacheorfitinmemory.Somovethosefieldstosourcequalifierandthenjoinwiththe
maintable.
14.Ifthereareseverallookupswiththesamedataset,thensharethecaches.
15.Ifwearegoingtoreturnonly1row,thenuseunconnectedlookup.
16.Alldataarereadintocacheintheorderthefieldsarelistedinlookupports.Ifwehaveanindexthatisevenpartiallyinthisorder,theloadingof
theselookupscanbespeededup.
17.Ifthetablethatweuseforlookuphasanindex(orifwehaveprivilegetoaddindextothetableinthedatabase,doso),thentheperformance
wouldincreasebothforcachedanduncachedlookups.
OptimizingtheBottlenecks
1. Ifthesourceisaflatfile,ensurethattheflatfileislocaltotheInformaticaserver.Ifsourceisarelationaltable,thentrynottousesynonymsor
aliases.
2. Ifthesourceisaflatfile,reducethenumberofbytes(Bydefaultitis1024bytesperline)theInformaticareadsperline.Ifwedothis,wecan
decreasetheLineSequentialBufferLengthsettingofthesessionproperties.
3. Ifpossible,giveaconditionalqueryinthesourcequalifiersothattherecordsarefilteredoffassoonaspossibleintheprocess.
4. Inthesourcequalifier,ifthequeryhasORDERBYorGROUPBY,thencreateanindexonthesourcetableandorderbytheindexfieldofthe
sourcetable.
PERFORMANCETUNINGOFTARGETS
Ifthetargetisaflatfile,ensurethattheflatfileislocaltotheInformaticaserver.Iftargetisarelationaltable,thentrynottousesynonymsoraliases.
1. Usebulkloadwheneverpossible.
2. Increasethecommitlevel.
3. Dropconstraintsandindexesofthetablebeforeloading.
PERFORMANCETUNINGOFMAPPINGS
Mappinghelpstochanneltheflowofdatafromsourcetotargetwithallthetransformationsinbetween.MappingistheskeletonofInformatica
loadingprocess.
1. Avoidexecutingmajorsqlqueriesfrommappletsormappings.
4/51
6/5/2015
2. Useoptimizedquerieswhenweareusingthem.
3. Reducethenumberoftransformationsinthemapping.Activetransformationslikerank,joiner,filter,aggregatoretcshouldbeusedaslessas
possible.
4. Removealltheunnecessarylinksbetweenthetransformationsfrommapping.
5. Ifasinglemappingcontainsmanytargets,thendividingthemintoseparatemappingscanimproveperformance.
6. Ifweneedtouseasinglesourcemorethanonceinamapping,thenkeeponlyonesourceandsourcequalifierinthemapping.Thencreate
differentdataflowsasrequiredintodifferenttargetsorsametarget.
7. Ifasessionjoinsmanysourcetablesinonesourcequalifier,thenanoptimizingquerywillimproveperformance.
8. InthesqlquerythatInformaticagenerates,ORDERBYwillbepresent.RemovetheORDERBYclauseifnotneededoratleastreducethe
numberofcolumnnamesinthatlist.Forbetterperformanceitisbesttoorderbytheindexfieldofthattable.
9. Combinethemappingsthatusesamesetofsourcedata.
10. Onamapping,fieldwiththesameinformationshouldbegiventhesametypeandlengththroughoutthemapping.Otherwisetimewillbespent
onfieldconversions.
11. Insteadofdoingcomplexcalculationinquery,useanexpressiontransformeranddothecalculationinthemapping.
12. Ifdataispassingthroughmultiplestagingareas,removingthestagingareawillincreaseperformance.
13. Storedproceduresreduceperformance.Trytokeepthestoredproceduressimpleinthemappings.
14. Unnecessarydatatypeconversionsshouldbeavoidedsincethedatatypeconversionsimpactperformance.
15. Transformationerrorsresultinperformancedegradation.Tryrunningthemappingafterremovingalltransformations.Ifitistaking
significantlylesstimethanwiththetransformations,thenwehavetofinetunethetransformation.
16. Keepdatabaseinteractionsaslessaspossible.
PERFORMANCETUNINGOFSESSIONS
Asessionspecifiesthelocationfromwherethedataistobetaken,wherethetransformationsaredoneandwherethedataistobeloaded.Ithas
variouspropertiesthathelpustoscheduleandrunthejobinthewaywewant.
1. Partitionthesession:Thiscreatesmanyconnectionstothesourceandtarget,andloadsdatainparallelpipelines.Eachpipelinewillbe
independentoftheother.Buttheperformanceofthesessionwillnotimproveifthenumberofrecordsisless.Alsotheperformancewillnot
improveifitdoesupdatesanddeletes.Sosessionpartitioningshouldbeusedonlyifthevolumeofdataishugeandthejobismainlyinsertion
ofdata.
2. Runthesessionsinparallelratherthanserialtogaintime,iftheyareindependentofeachother.
3. Dropconstraintsandindexesbeforewerunsession.Rebuildthemafterthesessionruncompletes.Droppingcanbedoneinpresessionscript
andRebuildinginpostsessionscript.Butifdataistoomuch,droppingindexesandthenrebuildingthemetc.willbenotpossible.Insuchcases,
stagealldata,precreatetheindex,useatransportabletablespaceandthenloadintodatabase.
4. Usebulkloading,externalloadingetc.Bulkloadingcanbeusedonlyifthetabledoesnothaveanindex.
5. InasessionwehaveoptionstoTreatrowsasDataDriven,Insert,UpdateandDelete.Ifupdatestrategiesareused,thenwehavetokeepitas
DataDriven.Butwhenthesessiondoesonlyinsertionofrowsintotargettable,ithastobekeptasInserttoimproveperformance.
6. Increasethedatabasecommitlevel(ThepointatwhichtheInformaticaserverissettocommitdatatothetargettable.Fore.g.commitlevelcan
5/51
6/5/2015
besetateveryevery50,000records)
7. Byavoidingbuiltinfunctionsasmuchaspossible,wecanimprovetheperformance.E.g.Forconcatenation,theoperator||isfasterthanthe
functionCONCAT().Souseoperatorsinsteadoffunctions,wherepossible.ThefunctionslikeIS_SPACES(),IS_NUMBER(),IFF(),
DECODE()etc.reducetheperformancetoabigextentinthisorder.Preferenceshouldbeintheoppositeorder.
8. Stringfunctionslikesubstring,ltrim,andrtrimreducetheperformance.Inthesources,usedelimitedstringsincasethesourceflatfilesoruse
varchardatatype.
9. ManipulatinghighprecisiondatatypeswillslowdownInformaticaserver.Sodisablehighprecision.
10. Localizeallsourceandtargettables,storedprocedures,views,sequencesetc.Trynottoconnectacrosssynonyms.Synonymsandaliasesslow
downtheperformance.
DATABASEOPTIMISATION
TogainthebestInformaticaperformance,thedatabasetables,storedproceduresandqueriesusedinInformaticashouldbetunedwell.
1. Ifthesourceandtargetareflatfiles,thentheyshouldbepresentinthesysteminwhichtheInformaticaserverispresent.
2. Increasethenetworkpacketsize.
3. TheperformanceoftheInformaticaserverisrelatedtonetworkconnections.Datagenerallymovesacrossanetworkatlessthan1MBper
second,whereasalocaldiskmovesdatafivetotwentytimesfaster.Thusnetworkconnectionsoftenaffectonsessionperformance.Soavoid
networkconnections.
4. Optimizetargetdatabases.
IDENTIFICATIONOFBOTTLENECKS
IDENTIFICATIONOFBOTTLENECKS
PerformanceofInformaticaisdependantontheperformanceofitsseveralcomponentslikedatabase,network,transformations,mappings,sessions
etc.TotunetheperformanceofInformatica,wehavetoidentifythebottleneckfirst.
Bottleneckmaybepresentinsource,target,transformations,mapping,session,databaseornetwork.Itisbesttoidentifyperformanceissuein
componentsintheordersource,target,transformations,mappingandsession.Afteridentifyingthebottleneck,applythetuningmechanismsin
whicheverwaytheyareapplicabletotheproject.
IdentifybottleneckinSource
Ifsourceisarelationaltable,putafiltertransformationinthemapping,justaftersourcequalifiermaketheconditionoffiltertoFALSE.Soall
recordswillbefilteredoffandnonewillproceedtootherpartsofthemapping.Inoriginalcase,withoutthetestfilter,totaltimetakenisasfollows:
TotalTime=timetakenby(source+transformations+targetload)
Nowbecauseoffilter,TotalTime=timetakenbysource
Soifsourcewasfine,theninthelattercase,sessionshouldtakelesstime.Stillifthesessiontakesnearequaltimeasformercase,thenthereisa
sourcebottleneck.
6/51
6/5/2015
IdentifybottleneckinTarget
Ifthetargetisarelationaltable,thensubstituteitwithaflatfileandrunthesession.Ifthetimetakennowisverymuchlessthanthetimetakenfor
thesessiontoloadtotable,thenthetargettableisthebottleneck.
IdentifybottleneckinTransformation
Removethetransformationfromthemappingandrunit.Notethetimetaken.Thenputthetransformationbackandrunthemappingagain.Ifthetime
takennowissignificantlymorethanprevioustime,thenthetransformationisthebottleneck.
Butremovaloftransformationfortestingcanbeapainforthedevelopersincethatmightrequirefurtherchangesforthesessiontogetintothe
workingmode.
SowecanputfilterwiththeFALSEconditionjustafterthetransformationandrunthesession.Ifthesessionruntakesequaltimewithandwithout
thistestfilter,thentransformationisthebottleneck.
Identifybottleneckinsessions
Wecanusethesessionlogtoidentifywhetherthesource,targetortransformationsaretheperformancebottleneck.Sessionlogscontainthread
summaryrecordslikethefollowing:
MASTER>PETL_24018Thread[READER_1_1_1]createdforthereadstageofpartitionpoint[SQ_test_all_text_data]hascompleted:TotalRun
Time=[11.703201]secs,TotalIdleTime=[9.560945]secs,BusyPercentage=[18.304876].
MASTER>PETL_24019Thread[TRANSF_1_1_1_1]createdforthetransformationstageofpartitionpoint[SQ_test_all_text_data]hascompleted:
TotalRunTime=[11.764368]secs,TotalIdleTime=[0.000000]secs,BusyPercentage=[100.000000].
Ifbusypercentageis100,thenthatpartisthebottleneck.
Basicallywehavetorelyonthreadstatisticstoidentifythecauseofperformanceissues.OncetheCollectPerformanceDataoption(Insession
Propertiestab)isenabled,alltheperformancerelatedinformationwouldappearinthelogcreatedbythesession.
PerformanceTuningOverview
PerformanceTuningOverview
Thegoalofperformancetuningistooptimizesessionperformancebyeliminatingperformancebottlenecks.Totunesessionperformance,first
identifyaperformancebottleneck,eliminateit,andthenidentifythenextperformancebottleneckuntilyouaresatisfiedwiththesessionperformance.
Youcanusethetestloadoptiontorunsessionswhenyoutunesessionperformance.
Ifyoutuneallthebottlenecks,youcanfurtheroptimizesessionperformancebyincreasingthenumberofpipelinepartitionsinthesession.Adding
partitionscanimproveperformancebyutilizingmoreofthesystemhardwarewhileprocessingthesession.
Becausedeterminingthebestwaytoimproveperformancecanbecomplex,changeonevariableatatime,andtimethesessionbothbeforeandafter
thechange.Ifsessionperformancedoesnotimprove,youmightwanttoreturntotheoriginalconfiguration.
Completethefollowingtaskstoimprovesessionperformance:
1. Optimizethetarget.EnablestheIntegrationServicetowritetothetargetsefficiently.
2. Optimizethesource.EnablestheIntegrationServicetoreadsourcedataefficiently.
3. Optimizethemapping.EnablestheIntegrationServicetotransformandmovedataefficiently.
7/51
6/5/2015
4. Optimizethetransformation.EnablestheIntegrationServicetoprocesstransformationsinamappingefficiently.
5. Optimizethesession.EnablestheIntegrationServicetorunthesessionmorequickly.
6. Optimizethegriddeployments.EnablestheIntegrationServicetorunonagridwithoptimalperformance.
7. OptimizethePowerCentercomponents.EnablestheIntegrationServiceandRepositoryServicetofunctionoptimally.
8. Optimizethesystem.EnablesPowerCenterserviceprocessestorunmorequickly.
TRANSFORMATIONSPART2ININFORMATICA
SQLTRANSFORMATION:
YoucanpassthedatabaseconnectioninformationtotheSQLtransformationasinputdataatruntime.ThetransformationprocessesexternalSQL
scriptsorSQLqueriesthatyoucreateinanSQLeditor.TheSQLtransformationprocessesthequeryandreturnsrowsanddatabaseerrors.
WhenyoucreateanSQLtransformation,youconfigurethefollowingoptions:
Mode:TheSQLtransformationrunsinoneofthefollowingmodes:
Scriptmode.TheSQLtransformationrunsANSISQLscriptsthatareexternallylocated.Youpassascriptnametothetransformationwith
eachinputrow.TheSQLtransformationoutputsonerowforeachinputrow.
Querymode.TheSQLtransformationexecutesaquerythatyoudefineinaqueryeditor.Youcanpassstringsorparameterstothequeryto
definedynamicqueriesorchangetheselectionparameters.YoucanoutputmultiplerowswhenthequeryhasaSELECTstatement.
Passiveoractivetransformation.TheSQLtransformationisanactivetransformationbydefault.Youcanconfigureitasapassive
transformationwhenyoucreatethetransformation.
Databasetype.ThetypeofdatabasetheSQLtransformationconnectsto.
Connectiontype.PassdatabaseconnectioninformationtotheSQLtransformationoruseaconnectionobject.
ScriptMode
AnSQLtransformationrunninginscriptmoderunsSQLscriptsfromtextfiles.YoupasseachscriptfilenamefromthesourcetotheSQL
transformationScriptNameport.Thescriptfilenamecontainsthecompletepathtothescriptfile.
Whenyouconfigurethetransformationtoruninscriptmode,youcreateapassivetransformation.Thetransformationreturnsonerowforeachinput
row.Theoutputrowcontainsresultsofthequeryandanydatabaseerror.
RulesandGuidelinesforScriptMode
UsethefollowingrulesandguidelinesforanSQLtransformationthatrunsinscriptmode:
Youcanuseastaticordynamicdatabaseconnectionwithscriptmode.
Toincludemultiplequerystatementsinascript,youcanseparatethemwithasemicolon.
Youcanusemappingvariablesorparametersinthescriptfilename.
8/51
6/5/2015
Thescriptcodepagedefaultstothelocaleoftheoperatingsystem.Youcanchangethelocaleofthescript.
ThescriptfilemustbeaccessiblebytheIntegrationService.TheIntegrationServicemusthavereadpermissionsonthedirectorythatcontains
thescript.
TheIntegrationServiceignorestheoutputofanySELECTstatementyouincludeintheSQLscript.TheSQLtransformationinscriptmode
doesnotoutputmorethanonerowofdataforeachinputrow.
YoucannotusescriptinglanguagessuchasOraclePL/SQLorMicrosoft/SybaseTSQLinthescript.
YoucannotusenestedscriptswheretheSQLscriptcallsanotherSQLscript.
Ascriptcannotacceptruntimearguments.
QueryMode
WhenyouconfiguretheSQLtransformationtoruninquerymode,youcreateanactivetransformation.
WhenanSQLtransformationrunsinquerymode,itexecutesanSQLquerythatyoudefineinthetransformation.
Youpassstringsorparameterstothequeryfromthetransformationinputportstochangethequerystatementorthequerydata.
YoucancreatethefollowingtypesofSQLqueriesintheSQLtransformation:
StaticSQLquery.Thequerystatementdoesnotchange,butyoucanusequeryparameterstochangethedata.TheIntegrationServiceprepares
thequeryonceandrunsthequeryforallinputrows.
DynamicSQLquery.Youcanchangethequerystatementsandthedata.TheIntegrationServicepreparesaqueryforeachinputrow.
RulesandGuidelinesforQueryMode
UsethefollowingrulesandguidelineswhenyouconfiguretheSQLtransformationtoruninquerymode:
ThenumberandtheorderoftheoutputportsmustmatchthenumberandorderofthefieldsinthequerySELECTclause.
Thenativedatatypeofanoutputportinthetransformationmustmatchthedatatypeofthecorrespondingcolumninthedatabase.The
IntegrationServicegeneratesarowerrorwhenthedatatypesdonotmatch.
WhentheSQLquerycontainsanINSERT,UPDATE,orDELETEclause,thetransformationreturnsdatatotheSQLErrorport,thepass
throughports,andtheNumRowsAffectedportwhenitisenabled.IfyouaddoutputportstheportsreceiveNULLdatavalues.
WhentheSQLquerycontainsaSELECTstatementandthetransformationhasapassthroughport,thetransformationreturnsdatatothepass
throughportwhetherornotthequeryreturnsdatabasedata.TheSQLtransformationreturnsarowwithNULLdataintheoutputports.
Youcannotaddthe"_output"suffixtooutputportnamesthatyoucreate.
YoucannotusethepassthroughporttoreturndatafromaSELECTquery.
WhenthenumberofoutputportsismorethanthenumberofcolumnsintheSELECTclause,theextraportsreceiveaNULLvalue.
WhenthenumberofoutputportsislessthanthenumberofcolumnsintheSELECTclause,theIntegrationServicegeneratesarowerror.
Youcanusestringsubstitutioninsteadofparameterbindinginaquery.However,theinputportsmustbestringdatatypes.
SQLTransformationProperties
9/51
6/5/2015
AfteryoucreatetheSQLtransformation,youcandefineportsandsetattributesinthefollowingtransformationtabs:
Ports.DisplaysthetransformationportsandattributesthatyoucreateontheSQLPortstab.
Properties.SQLtransformationgeneralproperties.
SQLSettings.AttributesuniquetotheSQLtransformation.
SQLPorts.SQLtransformationportsandattributes.
Note:YoucannotupdatethecolumnsonthePortstab.WhenyoudefineportsontheSQLPortstab,theydisplayonthePortstab.
PropertiesTab
ConfiguretheSQLtransformationgeneralpropertiesonthePropertiestab.SometransformationpropertiesdonotapplytotheSQLtransformationor
arenotconfigurable.
ThefollowingtabledescribestheSQLtransformationproperties:
Property
Description
RunTimeLocation
EnterapathrelativetotheIntegrationServicenodethatrunstheSQL
transformationsession.
Ifthispropertyisblank,theIntegrationServiceusestheenvironment
variabledefinedontheIntegrationServicenodetolocatetheDLLorshared
library.
YoumustcopyallDLLsorsharedlibrariestotheruntimelocationorto
theenvironmentvariabledefinedontheIntegrationServicenode.The
IntegrationServicefailstoloadtheprocedurewhenitcannotlocatethe
DLL,sharedlibrary,orareferencedfile.
Setstheamountofdetailincludedinthesessionlogwhenyourunasession
containingthistransformation.WhenyouconfiguretheSQLtransformation
tracingleveltoVerboseData,theIntegrationServicewriteseachSQL
queryitpreparestothesessionlog.
TracingLevel
IsPartitionable
Multiplepartitionsinapipelinecanusethistransformation.Usethe
followingoptions:
No.Thetransformationcannotbepartitioned.Thetransformationand
othertransformationsinthesamepipelinearelimitedtoonepartition.You
mightchooseNoifthetransformationprocessesalltheinputdatatogether,
suchasdatacleansing.
Locally.Thetransformationcanbepartitioned,buttheIntegrationService
mustrunallpartitionsinthepipelineonthesamenode.ChooseLocally
whendifferentpartitionsofthetransformationmustshareobjectsin
10/51
6/5/2015
UpdateStrategy
TransformationScope
OutputisRepeatable
GenerateTransaction
RequiresSingle
ThreadPerPartition
OutputisDeterministic
memory.
AcrossGrid.Thetransformationcanbepartitioned,andtheIntegration
Servicecandistributeeachpartitiontodifferentnodes.
DefaultisNo.
Thetransformationdefinestheupdatestrategyforoutputrows.Youcan
enablethispropertyforquerymodeSQLtransformations.
Defaultisdisabled.
ThemethodinwhichtheIntegrationServiceappliesthetransformation
logictoincomingdata.Usethefollowingoptions:
Row
Transaction
AllInput
Settransactionscopetotransactionwhenyouusetransactioncontrolin
staticquerymode.
DefaultisRowforscriptmodetransformations.DefaultisAllInputfor
querymodetransformations.
Indicatesiftheorderoftheoutputdataisconsistentbetweensessionruns.
Never.Theorderoftheoutputdataisinconsistentbetweensessionruns.
BasedOnInputOrder.Theoutputorderisconsistentbetweensessionruns
whentheinputdataorderisconsistentbetweensessionruns.
Always.Theorderoftheoutputdataisconsistentbetweensessionruns
eveniftheorderoftheinputdataisinconsistentbetweensessionruns.
DefaultisNever.
Thetransformationgeneratestransactionrows.Enablethispropertyfor
querymodeSQLtransformationsthatcommitdatainanSQLquery.
Defaultisdisabled.
IndicatesiftheIntegrationServiceprocesseseachpartitionofaprocedure
withonethread.
Thetransformationgenerateconsistentoutputdatabetweensessionruns.
Enablethispropertytoperformrecoveryonsessionsthatusethis
transformation.
Defaultisenabled.
CreateMapping:
Step1:Creatingaflatfileandimportingthesourcefromtheflatfile.
11/51
6/5/2015
CreateaNotepadandinitcreateatablebynamebikeswiththreecolumnsandthreerecordsinit.
Createonemorenotepadandnameitaspathforthebikes.InsidetheNotepadjusttypein(C:\bikes.txt)andsaveit.
Importthesource(secondnotepad)usingthesource>importfromthefile.Afterwhichwearegoannagetawizardwiththreesubsequent
windowsandfollowtheonscreeninstructionstocompletetheprocessofimportingthesource.
Step2:Importingthetargetandapplyingthetransformation.
Inthesamewayasspecifiedabovegotothetargets>importfromfileandselectanemptynotepadunderthenametargetforbikes(thisisonemore
blanknotepadwhichweshouldcreateandsaveundertheabovespecifiednameintheC:\).
Createtwocolumnsinthetargettableunderthenamereportanderror.
Weareallsethere.NowapplytheSQLtransformation.
InthefirstwindowwhenyouapplytheSQLtransformationweshouldselectthescriptmode.
ConnecttheSQtotheScriptNameunderinputsandconnecttheothertwofieldstotheoutputcorrespondingly.
Snapshotfortheabovediscussedthingsisgivenbelow.
12/51
6/5/2015
Step3:Designtheworkflowandrunit.
Createthetaskandtheworkflowusingthenamingconventions.
GotothemappingstabandclickontheSourceonthelefthandpanetospecifythepathfortheoutputfile.
Step4:Previewtheoutputdataonthetargettable.
================================================================
NORMALIZERTRANSFORMATION:
ActiveandConnectedTransformation.
TheNormalizertransformationnormalizesrecordsfromCOBOLandrelationalsources,allowingustoorganizethedata.
UseaNormalizertransformationinsteadoftheSourceQualifiertransformationwhenwenormalizeaCOBOLsource.
13/51
6/5/2015
WecanalsousetheNormalizertransformationwithrelationalsourcestocreatemultiplerowsfromasinglerowofdata.
Example1:Tocreate4recordsofeveryemployeeinEMPtable.
EMPwillbesourcetable.
CreatetargettableNormalizer_Multiple_Records.StructuresameasEMPanddatatypeofHIREDATEasVARCHAR2.
Createshortcutsasnecessary.
CreatingMapping:
1. Openfolderwherewewanttocreatethemapping.
2. ClickTools>MappingDesigner.
3. ClickMapping>Create>Givename.Ex:m_Normalizer_Multiple_Records
4. DragEMPandTargettable.
5. Transformation>Create>SelectExpression>Givename,Clickcreate,done.
6. PassallportsfromSQ_EMPtoExpressiontransformation.
7. Transformation>Create>SelectNormalizer>Givename,create&done.
8. TrydraggingportsfromExpressiontoNormalizer.NotPossible.
9. EditNormalizerandNormalizerTab.Addcolumns.ColumnsequaltocolumnsinEMPtableanddatatypealsosame.
10. NormalizerdoesnthaveDATETIMEdatatype.SoconvertHIREDATEtocharinexpressiont/f.Createoutputportout_hdateanddothe
conversion.
11. ConnectportsfromExpressiontoNormalizer.
12. EditNormalizerandNormalizerTab.AsEMPNOidentifiessourcerecordsandwewant4recordsofeveryemployee,giveOCCURfor
EMPNOas4.
13.
14. ClickApplyandthenOK.
15. Addlinkasshowninmappingbelow:
16. Mapping>Validate
17. Repository>Save
Makesessionandworkflow.
Giveconnectioninformationforsourceandtargettable.
Runworkflowandseeresult.
14/51
6/5/2015
Example2:Tobreakrowsintocolumns
Source:
Roll_Number Name
ENG
HINDI
MATHS
100
Amit
78
76
90
101
102
Rahul
Jessie
76
65
78
98
87
79
Target:
Roll_Number
Name
Marks
100
100
Amit
Amit
78
76
100
101
Amit
Rahul
90
76
101
101
Rahul
Rahul
78
87
102
102
Jessie
Jessie
65
98
102
Jessie
79
Makesourceasaflatfile.Importitandcreatetargettable.
CreateMappingasbefore.InNormalizertab,createonly3portsRoll_Number,NameandMarksasthereare3columnsintargettable.
Alsoaswehave3marksinsource,giveOccursas3forMarksinNormalizertab.
Connectaccordinglyandconnecttotarget.
ValidateandSave
MakeSessionandworkflowandRunit.GiveSourceFileDirectoryandSourceFilenameforsourceflatfileinsourcepropertiesinmapping
tabofsession.
Seetheresult.
====================================================================
SEQUENCEGENERATORTRANSFORMATION:
15/51
6/5/2015
PassiveandConnectedTransformation.
TheSequenceGeneratortransformationgeneratesnumericvalues.
UsetheSequenceGeneratortocreateuniqueprimarykeyvalues,replacemissingprimarykeys,orcyclethroughasequentialrangeofnumbers.
WeuseittogenerateSurrogateKeyinDWHenvironmentmostly.WhenwewanttoMaintainhistory,thenweneedakeyotherthanPrimaryKeyto
uniquelyidentifytherecord.SowecreateaSequence1,2,3,4andsoon.Weusethissequenceasthekey.Example:IfEMPNOisthekey,wecan
keeponlyonerecordintargetandcantmaintainhistory.SoweuseSurrogatekeyasPrimarykeyandnotEMPNO.
SequenceGeneratorPorts:
TheSequenceGeneratortransformationprovidestwooutputports:NEXTVALandCURRVAL.
Wecannoteditordeletetheseports.
Likewise,wecannotaddportstothetransformation.
NEXTVAL:
UsetheNEXTVALporttogeneratesequencenumbersbyconnectingittoaTransformationortarget.
Forexample,wemightconnectNEXTVALtotwotargettablesinamappingtogenerateuniqueprimarykeyvalues.
SequenceinTable1willbegeneratedfirst.Whentable1hasbeenloaded,onlythenSequencefortable2willbegenerated.
CURRVAL:
CURRVALisNEXTVALplustheIncrementByvalue.
WetypicallyonlyconnecttheCURRVALportwhentheNEXTVALportisAlreadyconnectedtoadownstreamtransformation.
IfweconnecttheCURRVALportwithoutconnectingtheNEXTVALport,theIntegrationServicepassesaconstantvalueforeachrow.
whenweconnecttheCURRVALportinaSequenceGeneratorTransformation,theIntegrationServiceprocessesonerowineachblock.
WecanoptimizeperformancebyconnectingonlytheNEXTVALportinaMapping.
16/51
6/5/2015
Example:TouseSequenceGeneratortransformation
EMPwillbesource.
CreateatargetEMP_SEQ_GEN_EXAMPLEinsharedfolder.StructuresameasEMP.AddtwomoreportsNEXT_VALUEand
CURR_VALUEtothetargettable.
Createshortcutsasneeded.
CreatingMapping:
1.Openfolderwherewewanttocreatethemapping.
2.ClickTools>MappingDesigner.
3.ClickMapping>Create>Givename.Ex:m_seq_gen_example
4.DragEMPandTargettable.
5.ConnectallportsfromSQ_EMPtotargettable.
6.Transformation>Create>SelectSequenceGeneratorforlist>Create>Done
7.ConnectNEXT_VALandCURR_VALfromSequenceGeneratortotarget.
8.ValidateMapping
9.Repository>Save
CreateSessionandthenworkflow.
Giveconnectioninformationforalltables.
Runworkflowandseetheresultintable.
17/51
6/5/2015
SequenceGeneratorProperties:
Setting
Required/Optional Description
StartValue
Required
StartvalueofthegeneratedsequencethatwewantIStouseifwe
useCycleoption.Defaultis0.
IncrementBy
Required
DifferencebetweentwoconsecutivevaluesfromtheNEXTVAL
port.
EndValue
Optional
MaximumvaluetheIntegrationServicegenerates.
CurrentValue
Optional
Firstvalueinthesequence.Ifcycleoptionused,thevaluemustbe
greaterthanorequaltothestartvalueandlesstheendvalue.
Cycle
Optional
Reset
Optional
Ifselected,theIntegrationServicecyclesthroughthesequence
range.Ex:StartValue:1EndValue10Sequencewillbefrom110
andagainstartfrom1.
Bydefault,lastvalueofsequenceduringsessionissavedto
repository.Nexttimethesequenceisstartedfromthevalued
saved.
Ifselected,theIntegrationServicegeneratesvaluesbasedonthe
originalcurrentvalueforeachsession.
PointstoPonder:
IfCurrentvalueis1andendvalue10,nocycleoption.Thereare17recordsinsource.Inthiscasesessionwillfail.
IfweconnectjustCURR_VALonly,thevaluewillbesameforallrecords.
IfCurrentvalueis1andendvalue10,cycleoptionthere.Startvalueis0.
Thereare17recordsinsource.Sequence:1210.0123
Tomakeabovesequenceas110120,giveStartValueas1.StartvalueisusedalongwithCycleoptiononly.
IfCurrentvalueis1andendvalue10,cycleoptionthere.Startvalueis1.
Thereare17recordsinsource.Sessionruns.11017.7willbesavedinrepository.Ifwerunsessionagain,sequencewillstartfrom8.
UseresetoptionifyouwanttostartsequencefromCURR_VALeverytime.
=====================================================================
AGGREGATORTRANSFORMATION:
18/51
6/5/2015
ConnectedandActiveTransformation
TheAggregatortransformationallowsustoperformaggregatecalculations,suchasaveragesandsums.
Aggregatortransformationallowsustoperformcalculationsongroups.
ComponentsoftheAggregatorTransformation
1. Aggregateexpression
2. Groupbyport
3. SortedInput
4. Aggregatecache
1)AggregateExpressions
Enteredinanoutputport.
Canincludenonaggregateexpressionsandconditionalclauses.
Thetransformationlanguageincludesthefollowingaggregatefunctions:
AVG,COUNT,MAX,MIN,SUM
FIRST,LAST
MEDIAN,PERCENTILE,STDDEV,VARIANCE
SingleLevelAggregateFunction:MAX(SAL)
NestedAggregateFunction:MAX(COUNT(ITEM))
NestedAggregateFunctions
InAggregatortransformation,therecanbemultiplesinglelevelfunctionsormultiplenestedfunctions.
AnAggregatortransformationcannothavebothtypesoffunctionstogether.
MAX(COUNT(ITEM))iscorrect.
MIN(MAX(COUNT(ITEM)))isnotcorrect.Itcanalsoincludeoneaggregatefunctionnestedwithinanotheraggregatefunction
ConditionalClauses
Wecanuseconditionalclausesintheaggregateexpressiontoreducethenumberofrowsusedintheaggregation.Theconditionalclausecanbeany
clausethatevaluatestoTRUEorFALSE.
SUM(COMMISSION,COMMISSION>QUOTA)
19/51
6/5/2015
NonAggregateFunctions
Wecanalsousenonaggregatefunctionsintheaggregateexpression.
IIF(MAX(QUANTITY)>0,MAX(QUANTITY),0))
2)GroupByPorts
Indicateshowtocreategroups.
Whengroupingdata,theAggregatortransformationoutputsthelastrowofeachgroupunlessotherwisespecified.
TheAggregatortransformationallowsustodefinegroupsforaggregations,ratherthanperformingtheaggregationacrossallinputdata.
Forexample,wecanfindMaximumSalaryforeveryDepartment.
InAggregatorTransformation,OpenPortstabandselectGroupByasneeded.
3)UsingSortedInput
Usetoimprovesessionperformance.
Tousesortedinput,wemustpassdatatotheAggregatortransformationsortedbygroupbyport,inascendingordescendingorder.
Whenweusethisoption,wetellAggregatorthatdatacomingtoitisalreadysorted.
WechecktheSortedInputOptioninPropertiesTabofthetransformation.
Iftheoptionischeckedbutwearenotpassingsorteddatatothetransformation,thenthesessionfails.
4)AggregatorCaches
ThePowerCenterServerstoresdataintheaggregatecacheuntilitcompletesAggregatecalculations.
Itstoresgroupvaluesinanindexcacheandrowdatainthedatacache.IfthePowerCenterServerrequiresmorespace,itstoresoverflow
valuesincachefiles.
Note:ThePowerCenterServerusesmemorytoprocessanAggregatortransformationwithsortedports.Itdoesnotusecachememory.Wedonot
needtoconfigurecachememoryforAggregatortransformationsthatusesortedports.
1)AggregatorIndexCache:
Theindexcacheholdsgroupinformationfromthegroupbyports.IfweareusingGroupByonDEPTNO,thenthiscachestoresvalues10,20,30etc.
AllGroupByColumnsareinAGGREGATORINDEXCACHE.Ex.DEPTNO
2)AggregatorDataCache:
DATACACHEisgenerallylargerthantheAGGREGATORINDEXCACHE.
20/51
6/5/2015
ColumnsinDataCache:
Variableportsifany
Nongroupbyinput/outputports.
Nongroupbyinputportsusedinnonaggregateoutputexpression.
Portcontainingaggregatefunction
1)Example:TocalculateMAX,MIN,AVGandSUMofsalaryofEMPtable.
EMPwillbesourcetable.
CreateatargettableEMP_AGG_EXAMPLEintargetdesigner.TableshouldcontainDEPTNO,MAX_SAL,MIN_SAL,AVG_SALand
SUM_SAL
Createtheshortcutsinyourfolder.
CreatingMapping:
21/51
6/5/2015
3.ClickMapping>Create>Givemappingname.Ex:m_agg_example
4.DragEMPfromsourceinmapping.
5.ClickTransformation>Create>SelectAGGREGATORfromlist.GivenameandclickCreate.Nowclickdone.
6.PassSALandDEPTNOonlyfromSQ_EMPtoAGGREGATORTransformation.
7.EditAGGREGATORTransformation.GotoPortsTab
8.Create4outputports:OUT_MAX_SAL,OUT_MIN_SAL,OUT_AVG_SAL,
OUT_SUM_SAL
9.OpenExpressionEditoronebyoneforalloutputportsandgivethe
calculations.Ex:MAX(SAL),MIN(SAL),AVG(SAL),SUM(SAL)
10.ClickApply>Ok.
11.Dragtargettablenow.
12.ConnecttheoutputportsfromRanktotargettable.
13.ClickMapping>Validate
14.Repository>Save
CreateSessionandWorkflowasdescribedearlier.RuntheWorkflowandseethedataintargettable.
Makesuretogiveconnectioninformationforalltables.
==================================================================
UNIONTRANSFORMATION:
ActiveandConnectedtransformation.
Uniontransformationisamultipleinputgrouptransformationthatyoucanusetomergedatafrommultiplepipelinesorpipelinebranchesintoone
pipelinebranch.ItmergesdatafrommultiplesourcessimilartotheUNIONALLSQLstatementtoCombinetheresultsfromtwoormoreSQL
statements.
UnionTransformationRulesandGuidelines
wecancreatemultipleinputgroups,butonlyoneoutputgroup.
wecanconnectheterogeneoussourcestoaUniontransformation.
allinputgroupsandtheoutputgroupmusthavematchingports.ThePrecision,datatype,andscalemustbeidenticalacrossallgroups.
TheUniontransformationdoesnotremoveduplicaterows.ToremoveDuplicaterows,wemustaddanothertransformationsuchasaRouteror
FilterTransformation.
wecannotuseaSequenceGeneratororUpdateStrategytransformationupstreamfromaUniontransformation.
UnionTransformationComponents
WhenweconfigureaUniontransformation,definethefollowingcomponents:
Transformationtab:Wecanrenamethetransformationandaddadescription.
22/51
6/5/2015
Propertiestab:Wecanspecifythetracinglevel.
Groupstab:Wecancreateanddeleteinputgroups.TheDesignerdisplaysgroupswecreateonthePortstab.
GroupPortstab:Wecancreateanddeleteportsfortheinputgroups.TheDesignerdisplaysportswecreateonthePortstab.
WecannotmodifythePorts,InitializationProperties,MetadataExtensions,orPortAttributeDefinitionstabsinaUniontransformation.
CreateinputgroupsontheGroupstab,andcreateportsontheGroupPortstab.WecancreateoneormoreinputgroupsontheGroupstab.The
Designercreatesoneoutputgroupbydefault.Wecannoteditordeletethedefaultoutputgroup.
Example:tocombinedataoftablesEMP_10,EMP_20andEMP_REST
ImporttablesEMP_10,EMP_20andEMP_RESTinsharedfolderinSources.
CreateatargettableEMP_UNION_EXAMPLEintargetdesigner.StructureshouldbesameEMPtable.
CreatingMapping:
23/51
6/5/2015
3. ClickMapping>Create>Givemappingname.Ex:m_union_example
4. DragEMP_10,EMP_20andEMP_RESTfromsourceinmapping.
5. ClickTransformation>Create>SelectUnionfromlist.GivenameandclickCreate.Nowclickdone.
6. PassportsfromSQ_EMP_10toUnionTransformation.
7. EditUnionTransformation.GotoGroupsTab
8. OnegroupwillbealreadythereaswedraggedportsfromSQ_DEPT_10toUnionTransformation.
9. Aswehave3sourcetables,we3need3inputgroups.Clickaddbuttontoadd2moregroups.SeeSampleMapping
10. Wecanalsomodifyportsinportstab.
11. ClickApply>Ok.
12. Dragtargettablenow.
13. ConnecttheoutputportsfromUniontotargettable.
14. ClickMapping>Validate
15. Repository>Save
Makesuretogiveconnectioninformationforall3sourceTables.
24/51
6/5/2015
Samplemappingpicture
=======================================================================
JOINERTRANSFORMATION:
ConnectedandActiveTransformation
UsedtojoinsourcedatafromtworelatedheterogeneoussourcesresidinginDifferentlocationsorfilesystems.Or,wecanjoindatafromthe
samesource.
Ifweneedtojoin3tables,thenweneed2JoinerTransformations.
TheJoinertransformationjoinstwosourceswithatleastonematchingport.TheJoinertransformationusesaconditionthatmatchesoneor
morepairsofPortsbetweenthetwosources.
Example:TojoinEMPandDEPTtables.
25/51
6/5/2015
EMPandDEPTwillbesourcetable.
CreateatargettableJOINER_EXAMPLEintargetdesigner.TableshouldContainallportsofEMPtableplusDNAMEandLOCasshown
below.
CreatingMapping:
3. ClickMapping>Create>Givemappingname.Ex:m_joiner_example
4. DragEMP,DEPT,andTarget.CreateJoinerTransformation.Linkasshownbelow.
5.SpecifythejoinconditioninConditiontab.Seestepsonnextpage.
6.SetMasterinPortstab.Seestepsonnextpage.
7.Mapping>Validate
8.Repository>Save.
26/51
6/5/2015
JOINCONDITION:
ThejoinconditioncontainsportsfrombothinputsourcesthatmustmatchforthePowerCenterServertojointworows.
Example:DEPTNO=DEPTNO1inabove.
1. EditJoinerTransformation>ConditionTab
2. Addcondition
Wecanaddasmanyconditionsasneeded.
Only=operatorisallowed.
IfwejoinCharandVarchardatatypes,thePowerCenterServercountsanyspacesthatpadCharvaluesaspartofthestring.Soifyoutrytojointhe
following:
Char(40)=abcdandVarchar(40)=abcd
ThentheCharvalueisabcdpaddedwith36blankspaces,andthePowerCenterServerdoesnotjointhetwofieldsbecausetheCharfieldcontains
trailingspaces.
Note:TheJoinertransformationdoesnotmatchnullvalues.
MASTERandDETAILTABLES
InJoiner,onetableiscalledasMASTERandotherasDETAIL.
MASTERtableisalwayscached.WecanmakeanytableasMASTER.
EditJoinerTransformation>PortsTab>SelectMforMastertable.
TablewithlessnumberofrowsshouldbemadeMASTERtoimprovePerformance.
Reason:
WhenthePowerCenterServerprocessesaJoinertransformation,itreadsrowsfrombothsourcesconcurrentlyandbuildstheindexanddata
cachebasedonthemasterrows.Sotablewithfewerrowswillbereadfastandcachecanbemadeastablewithmorerowsisstillbeingread.
Thefeweruniquerowsinthemaster,thefeweriterationsofthejoincomparisonoccur,whichspeedsthejoinprocess.
JOINERTRANSFORMATIONPROPERTIESTAB
CaseSensitiveStringComparison:Ifselected,thePowerCenterServerusescasesensitivestringcomparisonswhenperformingjoinson
stringcolumns.
CacheDirectory:Specifiesthedirectoryusedtocachemasterordetailrowsandtheindextotheserows.
JoinType:Specifiesthetypeofjoin:Normal,MasterOuter,DetailOuter,orFullOuter.
TracingLevel
JoinerDataCacheSize
27/51
6/5/2015
JoinerIndexCacheSize
SortedInput
JOINTYPES
InSQL,ajoinisarelationaloperatorthatcombinesdatafrommultipletablesintoasingleresultset.TheJoinertransformationactsinmuchthesame
manner,exceptthattablescanoriginatefromdifferentdatabasesorflatfiles.
TypesofJoins:
Normal
MasterOuter
DetailOuter
FullOuter
Note:Anormalormasterouterjoinperformsfasterthanafullouterordetailouterjoin.
Example:InEMP,wehaveemployeeswithDEPTNO10,20,30and50.InDEPT,wehaveDEPTNO10,20,30and40.DEPTwillbeMASTER
tableasithaslessrows.
NormalJoin:
Withanormaljoin,thePowerCenterServerdiscardsallrowsofdatafromthemasteranddetailsourcethatdonotmatch,basedonthecondition.
Allemployeesof10,20and30willbethereasonlytheyarematching.
MasterOuterJoin:
Thisjoinkeepsallrowsofdatafromthedetailsourceandthematchingrowsfromthemastersource.Itdiscardstheunmatchedrowsfromthemaster
source.
Alldataofemployeesof10,20and30willbethere.
TherewillbeemployeesofDEPTNO50andcorrespondingDNAMEandLOCColumnswillbeNULL.
DetailOuterJoin:
Thisjoinkeepsallrowsofdatafromthemastersourceandthematchingrowsfromthedetailsource.Itdiscardstheunmatchedrowsfromthedetail
source.
Allemployeesof10,20and30willbethere.
TherewillbeonerecordforDEPTNO40andcorrespondingdataofEMPcolumnswillbeNULL.
FullOuterJoin:
Afullouterjoinkeepsallrowsofdatafromboththemasteranddetailsources.
Alldataofemployeesof10,20and30willbethere.
28/51
6/5/2015
TherewillbeemployeesofDEPTNO50andcorrespondingDNAMEandLOCColumnswillbeNULL.
TherewillbeonerecordforDEPTNO40andcorrespondingdataofEMPColumnswillbeNULL.
USINGSORTEDINPUT
Usetoimprovesessionperformance.
tousesortedinput,wemustpassdatatotheJoinertransformationsortedbytheportsthatareusedinJoinCondition.
WechecktheSortedInputOptioninPropertiesTabofthetransformation.
IftheoptionischeckedbutwearenotpassingsorteddatatotheTransformation,thenthesessionfails.
WecanuseSORTERtosortdataorSourceQualifierincaseofRelationaltables.
JOINERCACHES
JoineralwayscachestheMASTERtable.Wecannotdisablecaching.ItbuildsIndexcacheandDataCachebasedonMASTERtable.
1)JoinerIndexCache:
AllColumnsofMASTERtableusedinJoinconditionareinJOINERINDEXCACHE.
Example:DEPTNOinourmapping.
2)JoinerDataCache:
MastercolumnnotinjoinconditionandusedforoutputtoothertransformationortargettableareinDataCache.
Example:DNAMEandLOCinourmappingexample.
PerformanceTuning:
Performjoinsinadatabasewhenpossible.
Joinsorteddatawhenpossible.
ForasortedJoinertransformation,designateasthemastersourcethesourcewithfewerduplicatekeyvalues.
Joinercan'tbeusedinfollowingconditions:
1. EitherinputpipelinecontainsanUpdateStrategytransformation.
2. WeconnectaSequenceGeneratortransformationdirectlybeforetheJoinertransformation.
=====================================================================
UpdateStraegyTRANSFORMATION:
ActiveandConnectedTransformation
29/51
6/5/2015
Tillnow,wehaveonlyinsertedrowsinourtargettables.Whatifwewanttoupdate,deleteorrejectrowscomingfromsourcebasedonsome
condition?
Example:IfAddressofaCUSTOMERchanges,wecanupdatetheoldaddressorkeepbotholdandnewaddress.Onerowisforoldandonefor
new.Thiswaywemaintainthehistoricaldata.
UpdateStrategyisusedwithLookupTransformation.InDWH,wecreateaLookupontargettabletodeterminewhetherarowalreadyexistsornot.
Thenweinsert,update,deleteorrejectthesourcerecordasperbusinessneed.
InPowerCenter,wesettheupdatestrategyattwodifferentlevels:
1. Withinasession
2. WithinaMapping
1.UpdateStrategywithinasession:
Whenweconfigureasession,wecaninstructtheIStoeithertreatallrowsinthesamewayoruseinstructionscodedintothesessionmappingtoflag
rowsfordifferentdatabaseoperations.
SessionConfiguration:
EditSession>Properties>TreatSourceRowsas:(Insert,Update,Delete,andDataDriven).Insertisdefault.SpecifyingOperationsforIndividual
TargetTables:
30/51
6/5/2015
Youcansetthefollowingupdatestrategyoptions:
Insert:Selectthisoptiontoinsertarowintoatargettable.
Delete:Selectthisoptiontodeletearowfromatable.
Update:Wehavethefollowingoptionsinthissituation:
UpdateasUpdate.Updateeachrowflaggedforupdateifitexistsinthetargettable.
UpdateasInsert.Inseteachrowflaggedforupdate.
UpdateelseInsert.Updatetherowifitexists.Otherwise,insertit.
Truncatetable:Selectthisoptiontotruncatethetargettablebeforeloadingdata.
2.FlaggingRowswithinaMapping
Withinamapping,weusetheUpdateStrategytransformationtoflagrowsforinsert,delete,update,orreject.
Operation
Constant
Numeric
Value
INSERT
DD_INSERT
0
31/51
6/5/2015
UPDATE
DD_UPDATE
1
DELETE
DD_DELETE
2
REJECT
DD_REJECT
3
UpdateStrategyExpressions:
Frequently,theupdatestrategyexpressionusestheIIForDECODEfunctionfromthetransformationlanguagetotesteachrowtoseeifitmeetsa
particularcondition.
IIF((ENTRY_DATE>APPLY_DATE),DD_REJECT,DD_UPDATE)
Or
IIF((ENTRY_DATE>APPLY_DATE),3,2)
TheaboveexpressioniswritteninPropertiesTabofUpdateStrategyT/f.
DDmeansDATADRIVEN
ForwardingRejectedRows:
WecanconfiguretheUpdateStrategytransformationtoeitherpassrejectedrowstothenexttransformationordropthem.
Steps:
1. CreateUpdateStrategyTransformation
2. Passallportsneededtoit.
3. SettheExpressioninPropertiesTab.
4. Connecttoothertransformationsortarget.
Performancetuning:
1. UseUpdateStrategytransformationaslessaspossibleinthemapping.
2. Donotuseupdatestrategytransformationifwejustwanttoinsertintotargettable,insteadusedirectmapping,directfilteringetc.
3. ForupdatingordeletingrowsfromthetargettablewecanuseUpdateStrategytransformationitself.
==================================================================
LookupTRANSFORMATION:
PassiveTransformation
CanbeConnectedorUnconnected.Dynamiclookupisconnected.
UseaLookuptransformationinamappingtolookupdatainaflatfileorarelationaltable,view,orsynonym.
WecanimportalookupdefinitionfromanyflatfileorrelationaldatabasetowhichboththePowerCenterClientandServercanconnect.
WecanusemultipleLookuptransformationsinamapping.
32/51
6/5/2015
ThePowerCenterServerqueriesthelookupsourcebasedonthelookupportsinthetransformation.ItcomparesLookuptransformationportvaluesto
lookupsourcecolumnvaluesbasedonthelookupcondition.Passtheresultofthelookuptoothertransformationsandatarget.
WecanusetheLookuptransformationtoperformfollowing:
Getarelatedvalue:EMPhasDEPTNObutDNAMEisnotthere.WeuseLookuptogetDNAMEfromDEPTtablebasedonLookup
Condition.
Performacalculation:WewantonlythoseEmployeeswhosSAL>Average(SAL).WewillwriteLookupOverridequery.
Updateslowlychangingdimensiontables:Mostimportantuse.WecanuseaLookuptransformationtodeterminewhetherrowsalreadyexist
inthetarget.
1.LOOKUPTYPES
WecanconfiguretheLookuptransformationtoperformthefollowingtypesoflookups:
ConnectedorUnconnected
RelationalorFlatFile
CachedorUncached
RelationalLookup:
WhenwecreateaLookuptransformationusingarelationaltableasalookupsource,wecanconnecttothelookupsourceusingODBCandimportthe
tabledefinitionasthestructurefortheLookuptransformation.
WecanoverridethedefaultSQLstatementifwewanttoaddaWHEREclauseorquerymultipletables.
Wecanuseadynamiclookupcachewithrelationallookups.
FlatFileLookup:
Whenweuseaflatfileforalookupsource,wecanuseanyflatfiledefinitionintherepository,orwecanimportit.Whenweimportaflatfilelookup
source,theDesignerinvokestheFlatFileWizard.
CachedorUncachedLookup:
WecanchecktheoptioninPropertiesTabtoCachetolookupornot.Bydefault,lookupiscached.
ConnectedandUnconnectedLookup
ConnectedLookup
UnconnectedLookup
Receivesinputvaluesdirectlyfromthepipeline.
Receivesinputvaluesfromtheresultofa:LKP
expressioninanothertransformation.
Wecanuseadynamicorstaticcache.
Wecanuseastaticcache.
Cacheincludesalllookupcolumnsusedinthe
mapping.
Cacheincludesalllookup/outputportsinthelookup
conditionandthelookup/returnport.
33/51
6/5/2015
Ifthereisnomatchforthelookupcondition,the
PowerCenterServerreturnsthedefaultvalueforall
outputports.
Ifthereisamatchforthelookupcondition,the
PowerCenterServerreturnstheresultofthelookup
conditionforalllookup/outputports.
Passmultipleoutputvaluestoanother
transformation.
Supportsuserdefineddefaultvalues
Ifthereisnomatchforthelookupcondition,the
PowerCenterServerreturnsNULL.
Ifthereisamatchforthelookupcondition,the
PowerCenterServerreturnstheresultofthelookup
conditionintothereturnport.
Passoneoutputvaluetoanothertransformation.
Doesnotsupportuserdefineddefaultvalues.
2.LOOKUPT/FCOMPONENTS
DefinethefollowingcomponentswhenweconfigureaLookuptransformationinamapping:
Lookupsource
Ports
Properties
Condition
34/51
6/5/2015
1.LookupSource:
Wecanuseaflatfileorarelationaltableforalookupsource.WhenwecreateaLookupt/f,wecanimportthelookupsourcefromthefollowing
locations:
Anyrelationalsourceortargetdefinitionintherepository
Anyflatfilesourceortargetdefinitionintherepository
AnytableorfilethatboththePowerCenterServerandClientmachinecanconnecttoThelookuptablecanbeasingletable,orwecanjoin
multipletablesinthesamedatabaseusingalookupSQLoverrideinPropertiesTab.
2.Ports:
Ports
I
O
L
LookupType Number
Needed
Description
Connected
Unconnected
Connected
Unconnected
Connected
Unconnected
Minimum1
InputporttoLookup.UsuallyportsusedforJoincondition
areInputports.
PortsgoingtoanothertransformationfromLookup.
Unconnected
1Only
Minimum1
Minimum1
3.PropertiesTab
Options
LookupType
LookupSQLOverride Relational
LookupTableName
Relational
LookupCaching
Enabled
LookupPolicyon
MultipleMatch
FlatFile,Relational
FlatFile,Relational
Lookupport.TheDesignerautomaticallyDesignateseach
columninthelookupsourceasalookup(L)andoutput
port(O).
Returnport.UseonlyinunconnectedLookupt/fonly.
Description
OverridesthedefaultSQLstatementtoquerythelookup
table.
Specifiesthenameofthetablefromwhichthe
transformationlooksupandcachesvalues.
IndicateswhetherthePowerCenterServercacheslookup
valuesduringthesession.
DetermineswhathappenswhentheLookuptransformation
findsmultiplerowsthatmatchthelookupcondition.
Options:UseFirstValueorUseLastValueorUseAny
ValueorReportError
35/51
6/5/2015
LookupCondition
Connection
Information
SourceType
FlatFile,Relational
Relational
DisplaysthelookupconditionyousetintheConditiontab.
Specifiesthedatabasecontainingthelookuptable.
FlatFile,Relational
Lookupisfromadatabaseorflatfile.
LookupCache
DirectoryName
LookupCache
Persistent
DynamicLookup
Cache
RecacheFrom
LookupSource
InsertElseUpdate
FlatFile,Relational
Locationwherecacheisbuild.
FlatFile,Relational
WhethertousePersistentCacheornot.
FlatFile,Relational
WhethertouseDynamicCacheornot.
FlatFile,Relational
LookupDataCache
Size
FlatFile,Relational
Torebuildcacheifcachesourcechangesandweareusing
PersistentCache.
Useonlywithdynamiccachingenabled.Appliestorows
enteringtheLookuptransformationwiththerowtypeof
insert.
DataCacheSize
LookupIndexCache
Size
CacheFileName
Prefix
FlatFile,Relational
IndexCacheSize
FlatFile,Relational
Useonlywithpersistentlookupcache.Specifiesthefile
nameprefixtousewithpersistentlookupcachefiles.
Relational
SomeotherpropertiesforFlatFilesare:
DatetimeFormat
ThousandSeparator
DecimalSeparator
CaseSensitiveStringComparison
NullOrdering
SortedInput
4:ConditionTab
WeentertheLookupCondition.ThePowerCenterServerusesthelookupconditiontotestincomingvalues.Wecomparetransformationinputvalues
36/51
6/5/2015
withvaluesinthelookupsourceorcache,representedbylookupports.
Thedatatypesinaconditionmustmatch.
Whenweentermultipleconditions,thePowerCenterServerevaluateseachconditionasanAND,notanOR.
ThePowerCenterServermatchesnullvalues.
Theinputvaluemustmeetallconditionsforthelookuptoreturnavalue.
=,>,<,>=,<=,!=Operatorscanbeused.
Example:IN_DEPTNO=DEPTNO
In_DNAME='DELHI'
Tip:Ifweincludemorethanonelookupcondition,placetheconditionswithanequalsignfirsttooptimizelookupperformance.
Note:
1.Wecanuse=operatorincaseofDynamicCache.
2.ThePowerCenterServerfailsthesessionwhenitencountersmultiplekeysforaLookuptransformationconfiguredtouseadynamiccache.
3.ConnectedLookupTransformation
Example:TocreateaconnectedLookupTransformation
EMPwillbesourcetable.DEPTwillbeLOOKUPtable.
CreateatargettableCONN_Lookup_EXAMPLEintargetdesigner.TableshouldcontainallportsofEMPtableplusDNAMEandLOCas
shownbelow.
CreatingMapping:
3.ClickMapping>Create>Givename.Ex:m_CONN_LOOKUP_EXAMPLE
5.ConnectallfieldsfromSQ_EMPtotargetexceptDNAMEandLOC.
6.Transformation>Create>SelectLOOKUPfromlist.Givenameandclick
37/51
6/5/2015
Create.
7.TheFollowingscreenisdisplayed.
8.AsDEPTistheSourcedefinition,clickSourceandthenSelectDEPT.
9.ClickOk.
10.NowPassDEPTNOfromSQ_EMPtothisLookup.DEPTNOfromSQ_EMPwillbenamedasDEPTNO1.EditLookupandrenameitto
IN_DEPTNOinportstab.
11.NowgotoCONDITIONtabandaddCONDITION.
DEPTNO=IN_DEPTNOandClickApplyandthenOK.
Linkthemappingasshownbelow:
12.WearenotpassingIN_DEPTNOandDEPTNOtoanyothertransformationfromLOOKUPwecaneditthelookuptransformationandremove
theOUTPUTcheckfromthem.
13.Mapping>Validate
14.Repository>Save
CreateSessionandWorkflowasdescribedearlier.Runtheworkflowandseethedataintargettable.
MakesuretogiveconnectionforLOOKUPTablealso.
WeuseConnectedLookupwhenweneedtoreturnmorethanonecolumnfromLookuptable.ThereisnouseofReturnPortinConnectedLookup.
38/51
6/5/2015
SEEPROPERTYTABFORADVANCEDSETTINGS
4.UnconnectedLookupTransformation
AnunconnectedLookuptransformationisseparatefromthepipelineinthemapping.Wewriteanexpressionusingthe:LKPreferencequalifierto
callthelookupwithinanothertransformation.
StepstoconfigureUnconnectedLookup:
1. Addinputports.
2. Addthelookupcondition.
3. Designateareturnvalue.
4. Callthelookupfromanothertransformation.
Example:TocreateaunconnectedLookupTransformation
EMPwillbesourcetable.DEPTwillbeLOOKUPtable.
CreateatargettableUNCONN_Lookup_EXAMPLEintargetdesigner.TableshouldcontainallportsofEMPtableplusDNAMEasshown
below.
CreatingMapping:
3.ClickMapping>Create>Givename.Ex:m_UNCONN_LOOKUP_EXAMPLE
39/51
6/5/2015
5.NowTransformation>Create>SelectEXPRESSIONfromlist.Givename
andclickCreate.ThenClickDone.
6.PassallportsfromSQ_EMPtoEXPRESSIONtransformation.
7.ConnectallfieldsfromEXPRESSIONtotargetexceptDNAME.
8.Transformation>Create>SelectLOOKUPfromlist.Givenameandclick
Create.
9.FollowthestepsasinConnectedabovetocreateLookuponDEPTtable.
10.ClickOk.
11.NowEdittheLookupTransformation.GotoPortstab.
12.AsDEPTNOiscommoninsourceandLookup,createaportIN_DEPTNO
portstab.MakeitInputportonlyandGiveDatatypesameasDEPTNO.
13.DesignateDNAMEasReturnPort.CheckonRtomakeit.
40/51
6/5/2015
14.NowaddaconditioninConditionTab.
DEPTNO=IN_DEPTNOandClickApplyandthenOK.
15.NowweneedtocallthisLookupfromExpressionTransformation.
16.EditExpressiont/fandcreateanewoutputportout_DNAMEofdatatypeasDNAME.OpentheExpressioneditorandcallLookupasgiven
below:
WedoubleclickUnconninbottomofFunctionstabandasweneedonly
DEPTNO,wepassonlyDEPTNOasinput.
17.ValidatethecallinExpressioneditorandClickOK.
18.Mapping>Validate
19.RepositorySave.
MakesuretogiveconnectionforLOOKUPTablealso.
5.LookupCaches
WecanconfigureaLookuptransformationtocachethelookuptable.TheIntegrationService(IS)buildsacacheinmemorywhenitprocessesthe
firstrowofdatainacachedLookuptransformation.
TheIntegrationServicealsocreatescachefilesbydefaultinthe$PMCacheDir.Ifthedatadoesnotfitinthememorycache,theISstorestheoverflow
valuesinthecachefiles.Whensessioncompletes,ISreleasescachememoryanddeletesthecachefiles.
Ifweuseaflatfilelookup,theISalwayscachesthelookupsource.
41/51
6/5/2015
WesettheCachetypeinLookupProperties.
LookupCacheFiles
1.LookupIndexCache:
Storesdataforthecolumnsusedinthelookupcondition.
2.LookupDataCache:
ForaconnectedLookuptransformation,storesdatafortheconnectedoutputports,notincludingportsusedinthelookupcondition.
ForanunconnectedLookuptransformation,storesdatafromthereturnport.
TypesofLookupCaches:
1.StaticCache
Bydefault,theIScreatesastaticcache.ItcachesthelookupfileortableandLooksupvaluesinthecacheforeachrowthatcomesintothe
transformation.TheISdoesnotupdatethecachewhileitprocessestheLookuptransformation.
2.DynamicCache
Tocacheatargettableorflatfilesourceandinsertnewrowsorupdateexistingrowsinthecache,useaLookuptransformationwithadynamiccache.
TheISdynamicallyinsertsorupdatesdatainthelookupcacheandpassesdatatothetarget.Targettableisalsoourlookuptable.Nogoodfor
performanceiftableishuge.
3.PersistentCache
Ifthelookuptabledoesnotchangebetweensessions,wecanconfiguretheLookuptransformationtouseapersistentlookupcache.
TheISsavesandreusescachefilesfromsessiontosession,eliminatingthetimeRequiredtoreadthelookuptable.
4.RecachefromSource
Ifthepersistentcacheisnotsynchronizedwiththelookuptable,wecanConfiguretheLookuptransformationtorebuildthelookupcache.IfLookup
tablehaschanged,wecanusethistorebuildthelookupcache.
5.SharedCache
Unnamedcache:WhenLookuptransformationsinamappinghavecompatiblecachingstructures,theISsharesthecachebydefault.Youcan
onlysharestaticunnamedcaches.
Namedcache:Useapersistentnamedcachewhenwewanttoshareacachefileacrossmappingsorshareadynamicandastaticcache.The
cachingstructuresmustmatchorbecompatiblewithanamedcache.Youcansharestaticanddynamicnamedcaches.
42/51
6/5/2015
BuildingConnectedLookupCaches
Wecanconfigurethesessiontobuildcachessequentiallyorconcurrently.
Whenwebuildsequentialcaches,theIScreatescachesasthesourcerowsentertheLookuptransformation.
Whenweconfigurethesessiontobuildconcurrentcaches,theISdoesnotwaitforthefirstrowtoentertheLookuptransformationbeforeit
createscaches.Instead,itbuildsmultiplecachesconcurrently.
1.BuildingLookupCachesSequentially:
2.BuildingLookupCachesConcurrently:
Toconfigurethesessiontocreateconcurrentcaches
EditSession>InConfigObjectTab>AdditionalConcurrentPipelinesfor
LookupCacheCreation>Giveavaluehere(AutoByDefault)
Note:TheISbuildscachesforunconnectedLookupssequentiallyonly
43/51
6/5/2015
====================================================================
ExpressionTRANSFORMATION:
Passiveandconnectedtransformation.
UsetheExpressiontransformationtocalculatevaluesinasinglerowbeforewewritetothetarget.Forexample,wemightneedtoadjustemployee
salaries,concatenatefirstandlastnames,orconvertstringstonumbers.
UsetheExpressiontransformationtoperformanynonaggregatecalculations.
Example:Addition,Subtraction,Multiplication,Division,Concat,Uppercaseconversion,lowercaseconversionetc.
WecanalsousetheExpressiontransformationtotestconditionalstatementsbeforeweoutputtheresultstotargettablesorothertransformations.
Example:IF,Then,Decode
Thereare3typesofportsinExpressionTransformation:
Input
Output
Variable:Usedtostoreanytemporarycalculation.
CalculatingValues:
TousetheExpressiontransformationtocalculatevaluesforasinglerow,wemustincludethefollowingports:
Inputorinput/outputportsforeachvalueusedinthecalculation:Forexample:TocalculateTotalSalary,weneedsalaryandcommission.
Outputportfortheexpression:Weenteroneexpressionforeachoutputport.Thereturnvaluefortheoutputportneedstomatchthereturn
valueoftheexpression.
WecanentermultipleexpressionsinasingleExpressiontransformation.Wecancreateanynumberofoutputportsinthetransformation.
Example:CalculatingTotalSalaryofanEmployee
ImportthesourcetableEMPinSharedfolder.Ifitisalreadythere,thendontimport.
Insharedfolder,createthetargettableEmp_Total_SAL.KeepallportsasinEMPtableexceptSalandCommintargettable.AddTotal_SAL
porttostorethecalculation.
Createthenecessaryshortcutsinthefolder.
44/51
6/5/2015
CreatingMapping:
3. ClickMapping>Create>Givemappingname.Ex:m_totalsal
4. DragEMPfromsourceinmapping.
5. ClickTransformation>Create>SelectExpressionfromlist.GivenameandclickCreate.Nowclickdone.
6. LinkportsfromSQ_EMPtoExpressionTransformation.
7. EditExpressionTransformation.AswedonotwantSalandCommintarget,removecheckfromoutputportforbothcolumns.
8. Nowcreateanewportout_Total_SAL.Makeitasoutputportonly.
9. ClickthesmallbuttonthatappearsintheExpressionsectionofthedialogboxandentertheexpressionintheExpressionEditor.
45/51
6/5/2015
10. EnterexpressionSAL+COMM.YoucanselectSALandCOMMfromPortstabinexpressioneditor.
11. ChecktheexpressionsyntaxbyclickingValidate.
12. ClickOK>ClickApply>ClickOk.
13. NowconnecttheportsfromExpressiontotargettable.
14. ClickMapping>Validate
15. Repository>Save
AsCOMMisnull,Total_SALwillbenullinmostcases.Nowopenyourmappingandexpressiontransformation.SelectCOMMport,InDefault
Valuegive0.Nowapplychanges.ValidateMappingandSave.
Refreshthesessionandvalidateworkflowagain.Runtheworkflowandseetheresultagain.
NowuseERRORinDefaultvalueofCOMMtoskiprowswhereCOMMisnull.
46/51
6/5/2015
Syntax:ERROR(Anymessagehere)
Similarly,wecanuseABORTfunctiontoabortthesessionifCOMMisnull.
Syntax:ABORT(Anymessagehere)
Makesuretodoubleclickthesessionafterdoinganychangesinmapping.Itwillpromptthatmappinghaschanged.ClickOKtorefreshthemapping.
Runworkflowaftervalidatingandsavingtheworkflow.
Performancetuning:
ExpressiontransformationisusedtoperformsimplecalculationsandalsotodoSourcelookups.
1. Useoperatorsinsteadoffunctions.
2. Minimizetheusageofstringfunctions.
3. Ifweuseacomplexexpressionmultipletimesintheexpressiontransformer,thenMakethatexpressionasavariable.Thenweneedtouseonly
thisvariableforallcomputations.
===================================================================
RouterTransformation:
Activeandconnectedtransformation.
ARoutertransformationissimilartoaFiltertransformationbecausebothtransformationsallowyoutouseaconditiontotestdata.AFilter
transformationtestsdataforoneconditionanddropstherowsofdatathatdonotmeettheCondition.However,aRoutertransformationtestsdatafor
oneormoreconditionsAndgivesyoutheoptiontorouterowsofdatathatdonotmeetanyoftheconditionstoadefaultoutputgroup.
Example:IfwewanttokeepemployeesofFrance,India,USin3differenttables,thenwecanuse3Filtertransformationsor1Router
transformation.
MappingAusesthreeFiltertransformationswhileMappingBproducesthesameresultwithoneRoutertransformation.
ARoutertransformationconsistsofinputandoutputgroups,inputandoutputports,groupfilterconditions,andpropertiesthatweconfigureinthe
Designer.
47/51
6/5/2015
48/51
6/5/2015
WorkingwithGroups
ARoutertransformationhasthefollowingtypesofgroups:
Input:TheGroupthatgetstheinputports.
Output:UserDefinedGroupsandDefaultGroup.WecannotmodifyordeleteOutputportsortheirproperties.
UserDefinedGroups:Wecreateauserdefinedgrouptotestaconditionbasedonincomingdata.Auserdefinedgroupconsistsofoutputportsand
agroupfilterCondition.WecancreateandedituserdefinedgroupsontheGroupstabwiththeDesigner.Createoneuserdefinedgroupforeach
conditionthatwewanttospecify.
TheDefaultGroup:TheDesignercreatesthedefaultgroupafterwecreateonenewuserdefinedgroup.TheDesignerdoesnotallowustoeditor
deletethedefaultgroup.Thisgroupdoesnothaveagroupfilterconditionassociatedwithit.IfalloftheconditionsevaluatetoFALSE,theISpasses
therowtothedefaultgroup.
Example:FilteringemployeesofDepartment10toEMP_10,Department20toEMP_20andresttoEMP_REST
SourceisEMPTable.
Create3targettablesEMP_10,EMP_20andEMP_RESTinsharedfolder.StructureshouldbesameasEMPtable.
CreatingMapping:
3.ClickMapping>Create>Givemappingname.Ex:m_router_example
4.DragEMPfromsourceinmapping.
5.ClickTransformation>Create>SelectRouterfromlist.Givenameand
ClickCreate.Nowclickdone.
6.PassportsfromSQ_EMPtoRouterTransformation.
7.EditRouterTransformation.GotoGroupsTab
8.ClicktheGroupstab,andthenclicktheAddbuttontocreateauserdefinedGroup.Thedefaultgroupiscreatedautomatically..
9.ClicktheGroupFilterConditionfieldtoopentheExpressionEditor.
10.Enteragroupfiltercondition.Ex:DEPTNO=10
11.ClickValidatetocheckthesyntaxoftheconditionsyouentered.
49/51
6/5/2015
12.CreateanothergroupforEMP_20.Condition:DEPTNO=20
13.TherestoftherecordsnotmatchingtheabovetwoconditionswillbepassedtoDEFAULTgroup.Seesamplemapping
14.ClickOK>ClickApply>ClickOk.
15.Nowconnecttheportsfromroutertotargettables.
16.ClickMapping>Validate
17.Repository>Save
Makesuretogiveconnectioninformationforall3targettables.
50/51
6/5/2015
SampleMapping:
DifferencebetweenRouterandFilter:
Wecannotpassrejecteddataforwardinfilterbutwecanpassitinrouter.RejecteddataisinDefaultGroupofrouter.
========================================================
51/51

Push Down Optimisation

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Push Down Optimisation

Hochgeladen von

Copyright:

Verfügbare Formate

6/5/2015

Das könnte Ihnen auch gefallen