DATASTAGE TRAINING

Date:-03/09/2012

Objective:-DataStage Backup is weekly activity tasks which will be perform every Monday after Production Batch run. Two types of Backup we are taking.
1. Configuration Backup 2. Code Backup 1. Configuration Backup: Configuration backup needs to be taken from 4 servers. They are:1. Production server (10.144.223.210) 2. QA server (10.147.109.205) 3. AT/TEST server (10.147.109.202) 4. DEV server (10.147.109.201) 1. Backup from Production server (10.144.223.210) Step1:- Logon to code backup server (10.144.223.40) as a datastage user. Then with the help of winscp tool using dsadm user logon Production server (10.144.223.210) Step2:- Go to /root/datastage/projects/ These are the four main projects inside the projects folder. a. DWH_EU_prd b. DW_INV_PRD c. EDI_prd d. SHAREreporting_prd From each projects we will copy two files for configuration backup. 1. uvodbc.config 2. DSparams So from here we will take 8 files, 2 files from each project. Step3:-We will put these files in the code backup server (10.144.223.40). Create a folder like Backup_YYYYMMDD inside D:\DataStage_configfiles\Prod and put all 8 files inside this Backup_YYYYMMDD with the following nomenclature :1. DWH_EU_ prd _ YYYYMMDD_ uvodbc.config 2. DWH_EU_ prd _ YYYYMMDD_ DSparams 3. DW_INV_PRD_ YYYYMMDD_ uvodbc.config

4. 5. 6. 7. 8.

DW_INV_PRD_ YYYYMMDD_ DSparams EDI_prd_ YYYYMMDD_ uvodbc.config EDI_prd_ YYYYMMDD_ DSparams SHAREreporting_prd_ YYYYMMDD_ uvodbc.config SHAREreporting_prd_ YYYYMMDD_ DSparams

Step4:- Go to /root/software/datastage/Ascential/datastage/DSEngine Copy 3 files for backup 1. .odbc.ini 2. dsenv 3. uvconfig Step5:- Go to D:\DataStage_configfiles\Prod on the code backup server (10.144.223.40). Create Backup_YYYYMMDD folder. Paste 1. .odbc_ YYYYMMDD.ini 2. dsenv_ YYYYMMDD 3. uvconfig_ YYYYMMDD Step6:- Go to /root/software/oracle/product/10.2.0/network/admin Copy 1. tnsnames.ora

Step7:- Go to D:\DataStage_configfiles\Prod on the code backup server (10.144.223.40). Create Backup_YYYYMMDD folder. Paste 1. tnsnames_ YYYYMMDD.ora

2. Backup from QA server (10.147.109.205) Step1:- Logon to code backup server (10.144.223.40) as a datastage user.

Logon QA server (10.147.109.205) using winscp tool as dsadm user. Step2:- Go to /root/datastage/projects/ These are the four main projects inside the projects folder. a. DWH_EU_prd b. DW_INV_PRD c. EDI_prd d. SHAREreporting_prd From each projects we will copy two files for configuration backup. 1. uvodbc.config 2. DSparams So from here we will take 8 files, 2 files from each project. Step3:-We will put these files in the code backup server (10.144.223.40). Create a folder like Backup_YYYYMMDD inside D:\DataStage_configfiles\Prod and put all 8 files inside this Backup_YYYYMMDD with the following nomenclature :1. 2. 3. 4. 5. 6. 7. 8. DWH_EU_ prd _ YYYYMMDD_ uvodbc.config DWH_EU_ prd _ YYYYMMDD_ DSparams DW_INV_PRD_ YYYYMMDD_ uvodbc.config DW_INV_PRD_ YYYYMMDD_ DSparams EDI_prd_ YYYYMMDD_ uvodbc.config EDI_prd_ YYYYMMDD_ DSparams SHAREreporting_prd_ YYYYMMDD_ uvodbc.config SHAREreporting_prd_ YYYYMMDD_ DSparams

Step4:- Go to /root/software/datastage/Ascential/datastage/DSEngine Copy 3 files for backup 1. .odbc.ini 2. dsenv 3. uvconfig Step5:- Go to D:\DataStage_configfiles\Prod on the code backup server (10.144.223.40). Create Backup_YYYYMMDD folder. Paste 1. .odbc_ YYYYMMDD.ini 2. dsenv_ YYYYMMDD 3. uvconfig_ YYYYMMDD

Step6:- Go to /root/software/oracle/product/10.2.0/network/admin Copy 1. tnsnames.ora

Step7:- Go to D:\DataStage_configfiles\Prod on the code backup server (10.144.223.40). Create Backup_YYYYMMDD folder. Paste 1. tnsnames_ YYYYMMDD.ora 3. Backup from AT/TEST server (10.147.109.202) Step1:- Logon to code backup server (10.144.223.40) as a datastage user. Use winscp tool to log on to AT/TEST server (10.147.109.202) as dsadm user.

Step2:- Go to /root/software/datastage/Ascential/datastage/DSEngine Copy 1) 2) 3) 4)

.odbc.ini dsenv uvconfig uvodbc.config

Step3:-Create a folder named Backup_YYYYMMDD and paste the copied files on server (10.144.223.40) at D:\DataStage_configfiles\AT/TEST Put the files inside this folder Backup_YYYYMMDD with the nomenclature as:-. 1) 2) 3) 4) .odbc_ YYYYMMDD.ini dsenv_ YYYYMMDD uvconfig_ YYYYMMDD uvodbc.config _ YYYYMMDD

Step4:- Go to /root/software/oracleadm/product/10.2.0/network/admin Copy 1. tnsnames.ora

Step5:- Create a folder named Backup_YYYYMMDD at D:\DataStage_configfiles\AT/TEST on the code backup server (10.144.223.40). Paste the file inside this folder Backup_YYYYMMDD with the nomenclature :tnsnames_ YYYYMMDD.ora

4. Backup from DEV server (10.147.109.201)

Step1:- Logon to code backup server (10.144.223.40) as a datastage user. Use winscp tool to log on to AT/TEST server (10.147.109.201) as dsadm user.

Step2:- Go to /root/software/datastage/Ascential/datastage/DSEngine Copy 1) 2) 3) 4)

.odbc.ini dsenv uvconfig uvodbc.config

Step3:-Create a folder named Backup_YYYYMMDD and paste the copied files on server (10.144.223.40) at D:\DataStage_configfiles\AT Put the files inside this folder Backup_YYYYMMDD with the nomenclature as:-. 1) 2) 3) 4) .odbc_ YYYYMMDD.ini dsenv_ YYYYMMDD uvconfig_ YYYYMMDD uvodbc.config _ YYYYMMDD

Step4:- Go to /root/software/oracleadm/product/10.2.0/network/admin Copy 1. tnsnames.ora

Step5:- Create a folder named Backup_YYYYMMDD at D:\DataStage_configfiles\AT on the code backup server (10.144.223.40). Paste the file inside this folder Backup_YYYYMMDD with the nomenclature:tnsnames_ YYYYMMDD.ora

Backup from Prod server in UNIX (10.147.109.210)

Logon through PUTTY into 10.147.109.210 using dsadm as a user. Put all the files inside this location /software/configbkup/ Goto this path /root/software/oracleadm/product/10.2.0/network/admin Type :- ls a List of all the files will come. Use this command to copy the file and put into desire location. Type:-c p tnsnames.ora /software/configbkup/tnsnames/tnsnames_YYYYMMDD.ora Goto this path:- /root/software/datastage/Ascential/datastage/DSEngine Type :-ls a List of all the files will come. Type:-c p .odbc.ini Type:-c p dsenv /software/configbkup/odbc/.odbcYYYYMMDD.ini /software/configbkup/ dsenv/dsenvYYYYMMDD

Type:-c p uvodbc.config /software/configbkup/ uvodbc/ uvodbc YYYYMMDD.config Type:-c p uvconfig /software/configbkup/ uvconfig/uvconfigYYYYMMDD

1. Code Backup from Prod server (10.147.109.210)

There are two types of code backup. 1. Manual code backup 2. Automated code backup

1. Manual code backup: - Logon to code backup server (10.144.223.40)

Step1:-Open Administrator client then click on Projects tab, select one project then click properties tab then click on the unprotected tab then ok.

Step2:-Same way unprotect the all 4 projects one by one before code back. Step3:-Open Manager Client, click on Export tab then click on the Datastage component tab, select
the path where you want put the backup file. Select the file

Required path is:-D: \Prod_Datastage_Code_Backups\ Backups_YYYYMMDD\ Here we are exporting only .dsx file one from each project. SHAREreporting_prd_ YYYYMMDD.dsx EDI_prd_ YYYYMMDD. dsx DW_INV_PRD_ YYYYMMDD. dsx DWH_EU_ prd _ YYYYMMDD. Dsx

Step4:-After code backup put the projects into protected mode.

1. Automated code backup:- Logon to code backup server (10.144.223.40)

Goto this path:- C:\Documents and setting\datastage\DS_Backup_xxx.bat Double click on the backup bat file it will run and take the backup. If you want to verify this backup goto C:\Prod_Datastage_Code_Backups\ExportedBackups\Prod_Backup. Here we can check the above code backup has been taken correctly or not.

Date:-04/09/2012

Introduction and Project Overview and server configuration

There are 2 DB servers. SHARE(Hosted on Oracle) SMART(Hosted on SIEBEL)

Mail Delivery List

Guy Allen- QA deployment manager Sarah Barker Prod deployment Manager Mark Woods & Rosario Datastage SME Hans & Robert Arnold - DBA

Stages in Batch run

Extraction -> 2. Staging -> 3. BIW -> 4. MV refresh -> 5. End ETL
All the sequences are linked with SeqDW_ETL_Main. We have to start this sequence first and it will automatically trigger other sequence. If there is any abort during the batch run, the status of the job will be abort. The sequence related to concerned job will be in Abort/Restartable state (otherwise Running/Restartable). SeqDW_ETL_Main will also be in Abort/Restartable state if the sequence inside it will be in Abort/Restartable state. Once ETL Main is in Abort/Restartable state, Seqrecovery.InstanceName, which is DWH_ONE in production (Seqrecovery.DWH_ONE), will start automatically. Once Seqrecovery.DWH_ONE completed successfully, we can restart the batch run from SeqDW_ETL_Main if it is an approved error. If Seqrecovery.DWH_ONE fails then we have to reset the sequence and manually submit the job. Once seq.recovery completed we can submit SeqDW_ETL_Main. 1. Extraction-> Name of the sequence is SeqExtSrcToStg. It uploads the data from DB(Oracle and Seibel) to the AIX server 10.144.223.210 1st sequence to run. If there is any issue, then it should be looked into by E-Team(Oracle DB issue) and ES DBA team (Siebel DBA Issue). carol.lie@ricoh-europe.com is heading E-team. e-team@ricoh-europe.com is the DL Schemas present in this stage is Oracle_DWH and Siebel_DWH. It takes approximately 1 hr. for the extraction to complete. 2. Staging -> Name of the sequence is SeqStgToBDW. It uploads data from AIX server (10.144.223.210) to BDW database. First the comparison takes place between data already present in CDC tables and the extracted data. CDC tables are in staging_once schema. DCD tables ID are in CDC_Lookup and are also is in staging_one schema. Schema names are same in QA. BDW_ONE(used for DW_EU_PRD project) and BDW_01(used for DW_ENV_PROD) schemas are also present in BDW database. It takes approximately 1:30 Hrs. to complete.

3. BIW -> Name of the sequence is SeqBDWtoBIW. There are 2 sequences that run inside this sequence. o Load Dimension (seqLoadDimension). This sequence runs first as it holds the primary key to the fact tables. It takes 10-15 minutes to complete this sequence o Load Facts (seqLoadFact). This will update the fact tables. It takes 4-5 hours to complete the sequence. o This database is being looked in to by Oracle DBA. 4. Refresh MV -> Materialized views are being updated in this stage. Materialized views are stored physically on the server. It takes 30 minutes to complete this stage. Cube build starts after this stage. 5. End_ETL -> Name of the sequence is SeqEndETL. It takes 5hours to complete this sequence. No SLA for this but we have to monitor this. This is a house-keeping job which clears dataset created in the scratch space of application server (10.144.223.210). Datasets contains temporary data created in different stages of batch run. AnalyzeDBSchema is a job which runs in SeqEndETL. It gathers the table data. Projects in Datastage: DW_EU_PROD This is the only project with SLA. EDE_PRD o 1st week of the month after batch run, mostly after 3:00PM BST. o Ticket needs to be assigned to DWH queue before starting with the project. DW_INV_PRD o Order and stock reconciliation are part of this project. o Order Starts at around 5:10 AM BST and stock reconciliation starts at 5:15AM BST from Monday to Friday. Share_Reporting o RICOH CAPITAL starts at 12AM & SAR starts at 4AM BST. o Runs from Monday to Friday. o Tactical Projects are part of this. o SLAM and SLA_CHM batch runs on Monday. o SLA and SLA_CH batch runs on 2nd of the month. We will run the weekly batch if requested by the client.

o o

Likewise PPGM runs on Monday and PPG runs on 1st day of every month. If there is any there is clash between the dates of monthly and weekly run, then we have to deschedule the weekly run and only run the monthly run.

***Note :- FG500 is a sequence which runs once a month before 5th of every month. Its a part of DW_EU_PROD