DataStage PX Generic History Processing

DataStage PX History
Processing
Page 1 of 22
.
Table of Contents
1 Introduction........................................................................................................................................3
2 Generic PX History Processing.........................................................................................................4
2.1 Components...............................................................................................................................4
2.2 Using the Generic History Processing Sequence.......................................................................4
2.2.1 Overview............................................................................................................................4
2.2.2 Input Dataset......................................................................................................................4
2.2.3 Parameters..........................................................................................................................5
3 Working Examples.............................................................................................................................7
3.1 Overview....................................................................................................................................7
3.2 Example 1 SHP All Fields, Empty Target...............................................................................8
3.3 Example 2 SHP.....................................................................................................................10
3.4 Example 3 SHP Function D..................................................................................................13
3.5 Example 4 SHP, Multiple Days............................................................................................15
3.6 Example 5 RHP....................................................................................................................18
3.7 Example 6 EHP.....................................................................................................................21
Page 2 of 22
.
1 Introduction
History Processing has traditionally been handled within the UK Data Warehouse using a shared
container to perform change detection, and multiple extract, update and load jobs. The development of
these processes is a large portion of the overall development and testing effort.
A generic history processing sequence has therefore been developed that incorporates all of this
functionality and dramatically reduces the scope of development for inbound DataStage projects.
This document will describe the functionality and usage of the generic PX history processing sequence.
Page 3 of 22
.
1.1 Using the Generic History Processing Sequence
1.1.1 Overview
The generic PX history processing sequence caters for all of the commonly used history processing
variants. It also allows for pre change-detected feeds such as CDU that have a function field describing
the action to be taken for a particular record. The matrix below describes the actions that are possible
for each combination of processing type.
HPTYPE FUNCTION INSERT NEW UPDATE EXPIRE

SHP non
SHP IU
SHP D
RHP non
RHP IU
RHP D
EHP non
EHP IU
EHP D
Please note that not all of these combinations are sensible. For example, EHP with a function other than
D, or RHP with any function field set does not logically make sense.
1.1.2 Input Dataset
The generic PX history processing sequence requires a DataStage PX dataset as its input. The
following set of rules must be observed in order for the process to work correctly:
1. The dataset must exist in the temporary working directory pUKDWTEMPFILEPATH

2. The dataset should contain the following fields only:
REC_ACDT of type date

All the required key fields (names and data types must match the target table)
All the data fields that are supplied (names and data types must match the target
table)
An optional function field as on the CDU feeds. This field relates to the optional
function field and type parameters, and will contain either of the following characters
D, I or U. See parameters section for further information.
1.1.3 Parameters
The generic sequence requires the following parameters:
PARAMETER NAME DESCRIPTION

pUKDWDATABASE Database Name e.g. DWHL01
pUKDWUSERNAME Database User Name e.g. ukdw_liv
pUKDWPASSWORD Database User Password
pUKDWSCHEMA Database Schema Name e.g. DWHL001
pUKDWDATADATE Data Date
pUKDWTEMPFILEPATH The Working Directory
Page 4 of 22
.
pSUBJECT This is the subject name e.g. CDU03012
However, in some cases this may need to be derived slightly differently.
For example, if you were history processing data for multiple type/scheme
codes on the same table, you could put all the data together and include
type/scheme code as part of the key.
Alternatively, you could have multiple instances of the history processing
sequence for the same table but with different type/scheme codes, but here
you would have to adjust the value of pSUBJECT so that the job invocations
and dataset names remain unique.
e.g. pSUBJECT = pSUBJECT:'_':pTYPE_CODE
$APT_CONFIG_FILE $APT_CONFIG_FILE
$APT_DB2INSTANCE_HOME $APT_DB2INSTANCE_HOME
$DB2_SERVER $DB2_SERVER
$DB2_CLIENT_INSTANCE_NA $DB2_CLIENT_INSTANCE_NAME
ME
pTABLE_NAME The Name of the Target Table
pCLEANSEDDATASET pCLEANSEDDATASET
pSCHEMAPATH pSCHEMAPATH
pINPUTDATASET This is the name of the dataset created in the preceding transform job
pHPTYPE This can be either:
RHP (Refresh History Processing - this is where the input dataset is a full refresh
of the data).
SHP (Standard History Processing - this will be used most of the time)
EHP (Expire history processing only)
pKEYFIELD1 This is the name of the first key field e.g. ARRG_ID
pKEYFIELD1_TYPE This is the PX data type for the first key field e.g. int64 (BigInt)
pKEYFIELD2 This is the name of the second field if available. Set to '' (blank) if not.
pKEYFIELD2_TYPE Second key field data type, or '' (blank) if not required.
pKEYFIELD3 see above
pKEYFIELD3_TYPE see above
pNUMKEYS The number of key fields ( in this example, 1)
pOPTIONAL_FILTER_LOAD This is an optional additional filter for the invoked generic extract and load
processes within the history processing sequence. If the processing is scheme
code specific, this should be set to something like: "AND CLAS_SCHM_COD =
'????'"
pSELECT_COLS These are the fields to be extracted from the target table for history processing.
This should include:
1) The key fields
2) REC_ACDT
3) REC_EDT
4) The value fields in the input dataset
5) Any value fields from the target table that are not in the input dataset but need
to be fed forward for any new records.
pCMT_RD_FLG Y = Commited Reads, N = WITH UR

pFUNCFIELD If the input dataset has a function field containing "D,I,U" then this should be the
name of this field e.g. FUNCTION. If this is not the case then this should be set to
FUNCTION_DMY
pFUNC_TYPE This is the data type of the function field above. If the value above is
FUNCTION_DMY then this should be set to string[1]
Page 5 of 22
.
2 Working Examples
2.1 Overview
For the purpose of these examples, UK Data Warehouse table DWH_IP_ARRG_RELN will be used.
DWH_IP_ARRG_RELN looks as follows:
NAME DB2 TYPE DB2 LENGTH NULLS PX DATA TYPE

IP_ID BIGINT 8 No int64
ARRG_ID BIGINT 8 No int64
CLAS_SCHM_COD CHARACTER 8 No string[8]
REC_ACDT DATE 4 No date
REC_EDT DATE 4 No date
REC_TS TIMESTAMP 10 No timestamp[microseconds]
IP_ARRG_RELN_STDT DATE 4 Yes date
IP_ARRG_RELN_ENDT DATE 4 Yes date
CLAS_GRP_COD CHARACTER 8 No string[8]
SELF_CTFN_IND CHARACTER 1 Yes string[1]
ARRG_IP_SEQ_NO CHARACTER 1 Yes string[1]
The key fields here are defined as IP_ID, ARRG_ID and CLAS_SCHM_COD.
The values fields are defined as IP_ARRG_RELN_STDT, IP_ARRG_RELN_ENDT,

CLAS_GRP_COD, SELF_CTFN_IND and ARRG_IP_SEQ_NO.
Fields REC_ACDT, REC_EDT and REC_TS are special cases.
Dummy CLAS_SCHM_COD 9999 will be used.
Page 6 of 22
.
2.2 Example 1 SHP All Fields, Empty Target
Existing Data:
The target table (DWH_IP_ARRG_RELN) is empty for CLAS_SCHM_COD 9999.
Input data:
Input Data 1 (SHP)
IP_ID ARRG_ID CLAS_SCHM_COD REC_ACDT IP_ARRG_RELN_STDT IP_ARRG_RELN_ENDT CLAS_GRP_COD SELF_CTFN_IND ARRG_IP_SEQ_NO

111111111 111111111 9999 22/07/2008 22/07/2008 31/12/9999 300 X 1
222222222 222222222 9999 22/07/2008 22/07/2008 31/12/9999 300 X 1
333333333 333333333 9999 22/07/2008 22/07/2008 31/12/9999 300 X 1
444444444 444444444 9999 22/07/2008 22/07/2008 31/12/9999 300 X 1
555555555 555555555 9999 22/07/2008 22/07/2008 31/12/9999 300 X 1
666666666 666666666 9999 22/07/2008 22/07/2008 31/12/9999 300 X 1
777777777 777777777 9999 22/07/2008 22/07/2008 31/12/9999 300 X 1
888888888 888888888 9999 22/07/2008 22/07/2008 31/12/9999 300 X 1
Parameters:
pUKDWDATABASE = DWHU01
pUKDWUSERNAME = ukdw_liv
pUKDWPASSWORD = ********
pUKDWSCHEMA = DWHL001
pUKDWDATADATE = 2008-07-22
pUKDWTEMPFILEPATH = /ukdw/prd/troubleshoot/Rick
pSUBJECT = RIK00001
$APT_CONFIG_FILE = /DataStageProject/ukdmdev/Ascential/DataStage/Configurations/rick4node.apt
$APT_DB2INSTANCE_HOME = /home/ukdw_liv [$PROJDEF]
$DB2_SERVER = idwhu01 [$PROJDEF]
$DB2_CLIENT_INSTANCE_NAME = cdwhd32 [$PROJDEF]
pTABLE_NAME = DWH_IP_ARRG_RELN
pCLEANSEDDATASET = RIK00001CleansedDS
pSCHEMAPATH = /ukdw/prd/schemas
pINPUTDATASET = DWH_IP_ARRG_RELN_TEST_DATA.ds
Page 7 of 22
.
pHPTYPE = SHP
pKEYFIELD1 = IP_ID
pKEYFIELD1_TYPE = int64
pKEYFIELD2 = ARRG_ID
pKEYFIELD3 = CLAS_SCHM_COD
pKEYFIELD3_TYPE = string[8]
pKEYFIELD4 = ''
pKEYFIELD4_TYPE = ''
pKEYFIELD5 = ''
pKEYFIELD6 = ''
pNUMKEYS = 3
pOPTIONAL_FILTER_LOAD = AND CLAS_SCHM_COD = '9999'
pSELECT_COLS = IP_ID,ARRG_ID,CLAS_SCHM_COD,REC_ACDT,REC_EDT,IP_ARRG_RELN_STDT,IP_ARRG_RELN_ENDT,CLAS_GRP_COD,SELF_CTFN_IND,ARRG_IP_SEQ_NO
pCMT_RD_FLG = N
pFUNCFIELD = FUNCTION_DMY
pFUNC_TYPE = string[1]
Output Data:
Resulting Data 1
IP_ID ARRG_ID CLAS_SCHM REC_ACDT REC_EDT REC_TS IP_ARRG_R IP_ARRG_R CLAS_GR SELF_CT ARRG_IP_
_COD ELN_STDT ELN_ENDT P_COD FN_IND SEQ_NO
111111111 111111111 9999 22/07/2008 12/31/9999 2008-07-23-15.24.58.929555 22/07/2008 31/12/9999 300 X 1
222222222 222222222 9999 22/07/2008 12/31/9999 2008-07-23-15.24.58.957837 22/07/2008 31/12/9999 300 X 1
333333333 333333333 9999 22/07/2008 12/31/9999 2008-07-23-15.24.58.916477 22/07/2008 31/12/9999 300 X 1
444444444 444444444 9999 22/07/2008 12/31/9999 2008-07-23-15.24.59.003843 22/07/2008 31/12/9999 300 X 1
555555555 555555555 9999 22/07/2008 12/31/9999 2008-07-23-15.24.59.030621 22/07/2008 31/12/9999 300 X 1
666666666 666666666 9999 22/07/2008 12/31/9999 2008-07-23-15.24.58.871704 22/07/2008 31/12/9999 300 X 1
777777777 777777777 9999 22/07/2008 12/31/9999 2008-07-23-15.24.58.868768 22/07/2008 31/12/9999 300 X 1
888888888 888888888 9999 22/07/2008 12/31/9999 2008-07-23-15.24.58.981654 22/07/2008 31/12/9999 300 X 1
Page 8 of 22
.
2.3 Example 2 SHP
Existing Data:
The existing data is as per the output data section of example 1 above.
Input Data:
Input Data 2 (SHP)
IP_ID ARRG_ID CLAS_SCHM_COD REC_ACDT CLAS_GRP_COD SELF_CTFN_IND

111111111 111111111 9999 23/07/2008 400
222222222 222222222 9999 23/07/2008 400
333333333 333333333 9999 23/07/2008 400 Y
444444444 444444444 9999 23/07/2008 400 Y
555555555 555555555 9999 23/07/2008 400 X
666666666 666666666 9999 23/07/2008 400 N
777777777 777777777 9999 23/07/2008 400 X
888888888 888888888 9999 23/07/2008 400 N
Parameters:
pSUBJECT = RIK00001
pHPTYPE = SHP
Page 9 of 22
.
pKEYFIELD1 = IP_ID
pKEYFIELD4 = ''
pKEYFIELD5 = ''
pKEYFIELD6 = ''
pNUMKEYS = 3
pCMT_RD_FLG = N
Output Data: (N.B. affected fields / records are in green. Also, fields that are not specified in the input dataset are fed forward from the previous active record)
Resulting Data 2
IP_ID ARRG_ID CLAS_SCHM REC_ACDT REC_EDT REC_TS IP_ARRG_R IP_ARRG_R CLAS_GR SELF_CT ARRG_IP
_COD ELN_STDT ELN_ENDT P_COD FN_IND _SEQ_NO
111111111 111111111 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.916477 07/22/2008 12/31/9999 300 X 1
111111111 111111111 9999 23/07/2008 12/31/9999 2008-07-23-15.42.53.217026 07/22/2008 12/31/9999 400 1
222222222 222222222 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.929555 07/22/2008 12/31/9999 300 X 1
222222222 222222222 9999 23/07/2008 12/31/9999 2008-07-23-15.42.52.931065 07/22/2008 12/31/9999 400 1
333333333 333333333 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.957837 07/22/2008 12/31/9999 300 X 1
333333333 333333333 9999 23/07/2008 12/31/9999 2008-07-23-15.42.53.394934 07/22/2008 12/31/9999 400 Y 1
444444444 444444444 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.871704 07/22/2008 12/31/9999 300 X 1
444444444 444444444 9999 23/07/2008 12/31/9999 2008-07-23-15.42.53.372263 07/22/2008 12/31/9999 400 Y 1
555555555 555555555 9999 22/07/2008 23/07/2008 2008-07-23-15.24.59.003843 07/22/2008 12/31/9999 300 X 1
555555555 555555555 9999 23/07/2008 12/31/9999 2008-07-23-15.42.53.355138 07/22/2008 12/31/9999 400 X 1
666666666 666666666 9999 22/07/2008 23/07/2008 2008-07-23-15.24.59.030621 07/22/2008 12/31/9999 300 X 1
666666666 666666666 9999 23/07/2008 12/31/9999 2008-07-23-15.42.53.357872 07/22/2008 12/31/9999 400 N 1
777777777 777777777 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.868768 07/22/2008 12/31/9999 300 X 1
777777777 777777777 9999 23/07/2008 12/31/9999 2008-07-23-15.42.53.372831 07/22/2008 12/31/9999 400 X 1
Page 10 of 22
.
888888888 888888888 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.981654 07/22/2008 12/31/9999 300 X 1
888888888 888888888 9999 23/07/2008 12/31/9999 2008-07-23-15.42.52.984474 07/22/2008 12/31/9999 400 N 1
Page 11 of 22
.
2.4 Example 3 SHP Function D
Existing Data:
Input Data: (Note, a function field FUNCTION has been added here with the values set to D for delete/expire)
Input Data 3 (SHP - FUNCTION D)
IP_ID ARRG_ID CLAS_SCH REC_ACDT FUNCTION

M_COD
111111111 111111111 9999 24/07/2008 D
222222222 222222222 9999 24/07/2008 D
333333333 333333333 9999 24/07/2008 D
444444444 444444444 9999 24/07/2008 D
Parameters: (Note, parameter pFUNCFIELD has been set to the name of the additional function field FUNCTION)
pSUBJECT = RIK00001
pHPTYPE = SHP
pKEYFIELD1 = IP_ID
Page 12 of 22
.
pKEYFIELD4 = ''
pKEYFIELD5 = ''
pKEYFIELD6 = ''
pNUMKEYS = 3
pCMT_RD_FLG = N
pFUNCFIELD = FUNCTION
Output Data:
Resulting Data 3
111111111 111111111 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.916477 07/22/2008 12/31/9999 300 X 1
111111111 111111111 9999 23/07/2008 24/07/2008 2008-07-23-15.42.53.217026 07/22/2008 12/31/9999 400 1
222222222 222222222 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.929555 07/22/2008 12/31/9999 300 X 1
222222222 222222222 9999 23/07/2008 24/07/2008 2008-07-23-15.42.52.931065 07/22/2008 12/31/9999 400 1
333333333 333333333 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.957837 07/22/2008 12/31/9999 300 X 1
333333333 333333333 9999 23/07/2008 24/07/2008 2008-07-23-15.42.53.394934 07/22/2008 12/31/9999 400 Y 1
444444444 444444444 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.871704 07/22/2008 12/31/9999 300 X 1
444444444 444444444 9999 23/07/2008 24/07/2008 2008-07-23-15.42.53.372263 07/22/2008 12/31/9999 400 Y 1
555555555 555555555 9999 22/07/2008 23/07/2008 2008-07-23-15.24.59.003843 07/22/2008 12/31/9999 300 X 1
555555555 555555555 9999 23/07/2008 12/31/9999 2008-07-23-15.42.53.355138 07/22/2008 12/31/9999 400 X 1
666666666 666666666 9999 22/07/2008 23/07/2008 2008-07-23-15.24.59.030621 07/22/2008 12/31/9999 300 X 1
666666666 666666666 9999 23/07/2008 12/31/9999 2008-07-23-15.42.53.357872 07/22/2008 12/31/9999 400 N 1
777777777 777777777 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.868768 07/22/2008 12/31/9999 300 X 1
777777777 777777777 9999 23/07/2008 12/31/9999 2008-07-23-15.42.53.372831 07/22/2008 12/31/9999 400 X 1
888888888 888888888 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.981654 07/22/2008 12/31/9999 300 X 1
888888888 888888888 9999 23/07/2008 12/31/9999 2008-07-23-15.42.52.984474 07/22/2008 12/31/9999 400 N 1
Page 13 of 22
.
2.5 Example 4 SHP, Multiple Days
Existing Data:
Input Data: (Multiple days data for the same keys)
Input Data 4 (SHP)
IP_ID ARRG_ID CLAS_SCHM_COD REC_ACDT SELF_CTFN_IND

555555555 555555555 9999 24/07/2008 A
555555555 555555555 9999 25/07/2008 B
555555555 555555555 9999 26/07/2008 B
555555555 555555555 9999 27/07/2008 C
Parameters:
pSUBJECT = RIK00001
pHPTYPE = SHP
pKEYFIELD1 = IP_ID
Page 14 of 22
.
pKEYFIELD4 = ''
pKEYFIELD5 = ''
pKEYFIELD6 = ''
pNUMKEYS = 3
pCMT_RD_FLG = N
Output Data: (Note the record for the 26th has been ignored as there was no change from that of the 25th)
Resulting Data 4
111111111 111111111 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.916477 07/22/2008 12/31/9999 300 X 1
111111111 111111111 9999 23/07/2008 24/07/2008 2008-07-23-15.42.53.217026 07/22/2008 12/31/9999 400 1
222222222 222222222 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.929555 07/22/2008 12/31/9999 300 X 1
222222222 222222222 9999 23/07/2008 24/07/2008 2008-07-23-15.42.52.931065 07/22/2008 12/31/9999 400 1
333333333 333333333 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.957837 07/22/2008 12/31/9999 300 X 1
333333333 333333333 9999 23/07/2008 24/07/2008 2008-07-23-15.42.53.394934 07/22/2008 12/31/9999 400 Y 1
444444444 444444444 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.871704 07/22/2008 12/31/9999 300 X 1
444444444 444444444 9999 23/07/2008 24/07/2008 2008-07-23-15.42.53.372263 07/22/2008 12/31/9999 400 Y 1
555555555 555555555 9999 22/07/2008 23/07/2008 2008-07-23-15.24.59.003843 07/22/2008 12/31/9999 300 X 1
555555555 555555555 9999 23/07/2008 24/07/2008 2008-07-23-15.42.53.355138 07/22/2008 12/31/9999 400 X 1
555555555 555555555 9999 24/07/2008 25/07/2008 2008-07-29-09.01.53.019815 07/22/2008 12/31/9999 400 A 1
555555555 555555555 9999 25/07/2008 27/07/2008 2008-07-29-09.01.53.019619 07/22/2008 12/31/9999 400 B 1
555555555 555555555 9999 27/07/2008 12/31/9999 2008-07-29-09.01.52.954123 07/22/2008 12/31/9999 400 C 1
666666666 666666666 9999 22/07/2008 23/07/2008 2008-07-23-15.24.59.030621 07/22/2008 12/31/9999 300 X 1
666666666 666666666 9999 23/07/2008 12/31/9999 2008-07-23-15.42.53.357872 07/22/2008 12/31/9999 400 N 1
777777777 777777777 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.868768 07/22/2008 12/31/9999 300 X 1
777777777 777777777 9999 23/07/2008 12/31/9999 2008-07-23-15.42.53.372831 07/22/2008 12/31/9999 400 X 1
888888888 888888888 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.981654 07/22/2008 12/31/9999 300 X 1
888888888 888888888 9999 23/07/2008 12/31/9999 2008-07-23-15.42.52.984474 07/22/2008 12/31/9999 400 N 1
Page 15 of 22
.
Page 16 of 22
.
2.6 Example 5 RHP
Existing Data:
Input Data:
Input Data 5 (RHP)
IP_ID ARRG_ID CLAS_SCHM_COD REC_ACDT SELF_CTFN_IND

555555555 555555555 9999 28/07/2008 R
666666666 666666666 9999 28/07/2008 R
888888888 888888888 9999 28/07/2008 R
999999999 999999999 9999 28/07/2008 R
Parameters:
pUKDWDATABASE=DWHU01
pUKDWUSERNAME=ukdw_liv
pUKDWPASSWORD=LD:@09VA?93M0O4I=9J50KH6
pUKDWSCHEMA=DWHL001
pUKDWDATADATE=2008-07-28
pUKDWTEMPFILEPATH=/ukdw/prd/troubleshoot/Rick
pSUBJECT=RIK00001
$APT_CONFIG_FILE=/DataStageProject/ukdmdev/Ascential/DataStage/Configurations/rick4node.apt
$APT_DB2INSTANCE_HOME=/home/ukdw_liv
$DB2_SERVER=idwhu01
$DB2_CLIENT_INSTANCE_NAME=cdwhd32
pTABLE_NAME=DWH_IP_ARRG_RELN
pCLEANSEDDATASET=RIK00001CleansedDS
pSCHEMAPATH=/ukdw/prd/schemas
pINPUTDATASET=DWH_IP_ARRG_RELN_TEST_DATA.ds
pHPTYPE=RHP
pKEYFIELD1=IP_ID
pKEYFIELD1_TYPE=int64
pKEYFIELD2=ARRG_ID
pKEYFIELD3=CLAS_SCHM_COD
Page 17 of 22
.
pKEYFIELD3_TYPE=string[8]
pKEYFIELD4=''
pKEYFIELD4_TYPE=''
pKEYFIELD5=''
pKEYFIELD5_TYPE=''
pKEYFIELD6=''
pKEYFIELD6_TYPE=''
pNUMKEYS=3
pOPTIONAL_FILTER_LOAD=AND CLAS_SCHM_COD = '9999'
pSELECT_COLS=IP_ID,ARRG_ID,CLAS_SCHM_COD,REC_ACDT,REC_EDT,IP_ARRG_RELN_STDT,IP_ARRG_RELN_ENDT,CLAS_GRP_COD,SELF_CTFN_IND,ARRG_IP_SEQ_NO
pCMT_RD_FLG=N
pFUNCFIELD=FUNCTION_DMY
pFUNC_TYPE=string[1]
Output Data: (Note when using RHP, the absence of an active record in this case IP_ID 77777777 generates an expire record)
Resulting Data 5
111111111 111111111 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.916477 07/22/2008 12/31/9999 300 X 1
111111111 111111111 9999 23/07/2008 24/07/2008 2008-07-23-15.42.53.217026 07/22/2008 12/31/9999 400 1
222222222 222222222 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.929555 07/22/2008 12/31/9999 300 X 1
222222222 222222222 9999 23/07/2008 24/07/2008 2008-07-23-15.42.52.931065 07/22/2008 12/31/9999 400 1
333333333 333333333 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.957837 07/22/2008 12/31/9999 300 X 1
333333333 333333333 9999 23/07/2008 24/07/2008 2008-07-23-15.42.53.394934 07/22/2008 12/31/9999 400 Y 1
444444444 444444444 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.871704 07/22/2008 12/31/9999 300 X 1
444444444 444444444 9999 23/07/2008 24/07/2008 2008-07-23-15.42.53.372263 07/22/2008 12/31/9999 400 Y 1
555555555 555555555 9999 22/07/2008 23/07/2008 2008-07-23-15.24.59.003843 07/22/2008 12/31/9999 300 X 1
555555555 555555555 9999 23/07/2008 24/07/2008 2008-07-23-15.42.53.355138 07/22/2008 12/31/9999 400 X 1
555555555 555555555 9999 24/07/2008 25/07/2008 2008-07-29-09.01.53.019815 07/22/2008 12/31/9999 400 A 1
555555555 555555555 9999 25/07/2008 27/07/2008 2008-07-29-09.01.53.019619 07/22/2008 12/31/9999 400 B 1
555555555 555555555 9999 27/07/2008 28/07/2008 2008-07-29-09.01.52.954123 07/22/2008 12/31/9999 400 C 1
555555555 555555555 9999 28/07/2008 12/31/9999 2008-07-29-12.10.42.394491 07/22/2008 12/31/9999 400 R 1
666666666 666666666 9999 22/07/2008 23/07/2008 2008-07-23-15.24.59.030621 07/22/2008 12/31/9999 300 X 1
666666666 666666666 9999 23/07/2008 28/07/2008 2008-07-23-15.42.53.357872 07/22/2008 12/31/9999 400 N 1
666666666 666666666 9999 28/07/2008 12/31/9999 2008-07-29-12.10.42.439295 07/22/2008 12/31/9999 400 R 1
777777777 777777777 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.868768 07/22/2008 12/31/9999 300 X 1
777777777 777777777 9999 23/07/2008 28/07/2008 2008-07-23-15.42.53.372831 07/22/2008 12/31/9999 400 X 1
888888888 888888888 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.981654 07/22/2008 12/31/9999 300 X 1
Page 18 of 22
.
888888888 888888888 9999 23/07/2008 28/07/2008 2008-07-23-15.42.52.984474 07/22/2008 12/31/9999 400 N 1
888888888 888888888 9999 28/07/2008 12/31/9999 2008-07-23-15.42.52.984474 07/22/2008 12/31/9999 400 R 1
999999999 999999999 9999 28/07/2008 12/31/9999 2008-07-29-12.10.42.229393 R
Page 19 of 22
.
2.7 Example 6 EHP
Existing Data:
Input Data:
Input Data 6 (EHP)
IP_ID ARRG_ID CLAS_SCHM_COD REC_ACDT

555555555 555555555 9999 29/07/2008
666666666 666666666 9999 29/07/2008
888888888 888888888 9999 29/07/2008
999999999 999999999 9999 29/07/2008
Parameters:
pUKDWDATABASE=DWHU01
pUKDWUSERNAME=ukdw_liv
pUKDWPASSWORD=LD:@09VA?93M0O4I=9J50KH6
pUKDWSCHEMA=DWHL001
pUKDWDATADATE=2008-07-29
pUKDWTEMPFILEPATH=/ukdw/prd/troubleshoot/Rick
pSUBJECT=RIK00001
$APT_CONFIG_FILE=/DataStageProject/ukdmdev/Ascential/DataStage/Configurations/rick4node.apt
$APT_DB2INSTANCE_HOME=/home/ukdw_liv
$DB2_SERVER=idwhu01
$DB2_CLIENT_INSTANCE_NAME=cdwhd32
pTABLE_NAME=DWH_IP_ARRG_RELN
pCLEANSEDDATASET=RIK00001CleansedDS
pSCHEMAPATH=/ukdw/prd/schemas
pINPUTDATASET=DWH_IP_ARRG_RELN_TEST_DATA6.ds
pHPTYPE=EHP
pKEYFIELD1=IP_ID
pKEYFIELD2=ARRG_ID
pKEYFIELD3=CLAS_SCHM_COD
pKEYFIELD3_TYPE=string[8]
Page 20 of 22
.
pKEYFIELD4=''
pKEYFIELD4_TYPE=''
pKEYFIELD5=''
pKEYFIELD5_TYPE=''
pKEYFIELD6=''
pKEYFIELD6_TYPE=''
pNUMKEYS=3
pOPTIONAL_FILTER_LOAD=AND CLAS_SCHM_COD = '9999'
pSELECT_COLS=IP_ID,ARRG_ID,CLAS_SCHM_COD,REC_ACDT,REC_EDT,IP_ARRG_RELN_STDT,IP_ARRG_RELN_ENDT,CLAS_GRP_COD,SELF_CTFN_IND,ARRG_IP_SEQ_NO
pCMT_RD_FLG=N
pFUNCFIELD=FUNCTION_DMY
pFUNC_TYPE=string[1]
Output Data:
Resulting Data 6
111111111 111111111 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.916477 07/22/2008 12/31/9999 300 X 1
111111111 111111111 9999 23/07/2008 24/07/2008 2008-07-23-15.42.53.217026 07/22/2008 12/31/9999 400 1
222222222 222222222 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.929555 07/22/2008 12/31/9999 300 X 1
222222222 222222222 9999 23/07/2008 24/07/2008 2008-07-23-15.42.52.931065 07/22/2008 12/31/9999 400 1
333333333 333333333 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.957837 07/22/2008 12/31/9999 300 X 1
333333333 333333333 9999 23/07/2008 24/07/2008 2008-07-23-15.42.53.394934 07/22/2008 12/31/9999 400 Y 1
444444444 444444444 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.871704 07/22/2008 12/31/9999 300 X 1
444444444 444444444 9999 23/07/2008 24/07/2008 2008-07-23-15.42.53.372263 07/22/2008 12/31/9999 400 Y 1
555555555 555555555 9999 22/07/2008 23/07/2008 2008-07-23-15.24.59.003843 07/22/2008 12/31/9999 300 X 1
555555555 555555555 9999 23/07/2008 24/07/2008 2008-07-23-15.42.53.355138 07/22/2008 12/31/9999 400 X 1
555555555 555555555 9999 24/07/2008 25/07/2008 2008-07-29-09.01.53.019815 07/22/2008 12/31/9999 400 A 1
555555555 555555555 9999 25/07/2008 27/07/2008 2008-07-29-09.01.53.019619 07/22/2008 12/31/9999 400 B 1
555555555 555555555 9999 27/07/2008 28/07/2008 2008-07-29-09.01.52.954123 07/22/2008 12/31/9999 400 C 1
555555555 555555555 9999 28/07/2008 29/07/2008 2008-07-29-12.10.42.394491 07/22/2008 12/31/9999 400 R 1
666666666 666666666 9999 22/07/2008 23/07/2008 2008-07-23-15.24.59.030621 07/22/2008 12/31/9999 300 X 1
666666666 666666666 9999 23/07/2008 28/07/2008 2008-07-23-15.42.53.357872 07/22/2008 12/31/9999 400 N 1
666666666 666666666 9999 28/07/2008 29/07/2008 2008-07-29-12.10.42.439295 07/22/2008 12/31/9999 400 R 1
777777777 777777777 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.868768 07/22/2008 12/31/9999 300 X 1
777777777 777777777 9999 23/07/2008 28/07/2008 2008-07-23-15.42.53.372831 07/22/2008 12/31/9999 400 X 1
888888888 888888888 9999 22/07/2008 23/07/2008 2008-07-23-15.24.58.981654 07/22/2008 12/31/9999 300 X 1
Page 21 of 22
.
888888888 888888888 9999 23/07/2008 28/07/2008 2008-07-23-15.42.52.984474 07/22/2008 12/31/9999 400 N 1
888888888 888888888 9999 28/07/2008 29/07/2008 2008-07-23-15.42.52.984474 07/22/2008 12/31/9999 400 R 1
999999999 999999999 9999 28/07/2008 29/07/2008 2008-07-29-12.10.42.229393 R
Page 22 of 22
.

DataStage PX Generic History Processing

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

DataStage PX Generic History Processing

Hochgeladen von

Copyright:

Verfügbare Formate

DataStage PX History

HPTYPE FUNCTION INSERT NEW UPDATE EXPIRE

1.1.2 Input Dataset

1. The dataset must exist in the temporary working directory pUKDWTEMPFILEPATH

REC_ACDT of type date

The generic sequence requires the following parameters:

PARAMETER NAME DESCRIPTION

pCMT_RD_FLG Y = Commited Reads, N = WITH UR

DWH_IP_ARRG_RELN looks as follows:

NAME DB2 TYPE DB2 LENGTH NULLS PX DATA TYPE

The values fields are defined as IP_ARRG_RELN_STDT, IP_ARRG_RELN_ENDT,

Fields REC_ACDT, REC_EDT and REC_TS are special cases.

Dummy CLAS_SCHM_COD 9999 will be used.

The target table (DWH_IP_ARRG_RELN) is empty for CLAS_SCHM_COD 9999.

Input Data 1 (SHP)

IP_ID ARRG_ID CLAS_SCHM_COD REC_ACDT IP_ARRG_RELN_STDT IP_ARRG_RELN_ENDT CLAS_GRP_COD SELF_CTFN_IND ARRG_IP_SEQ_NO

Input Data 2 (SHP)

IP_ID ARRG_ID CLAS_SCHM_COD REC_ACDT CLAS_GRP_COD SELF_CTFN_IND

Input Data 3 (SHP - FUNCTION D)

IP_ID ARRG_ID CLAS_SCH REC_ACDT FUNCTION

Input Data: (Multiple days data for the same keys)

Input Data 4 (SHP)

IP_ID ARRG_ID CLAS_SCHM_COD REC_ACDT SELF_CTFN_IND

Input Data 5 (RHP)

IP_ID ARRG_ID CLAS_SCHM_COD REC_ACDT SELF_CTFN_IND

Input Data 6 (EHP)

IP_ID ARRG_ID CLAS_SCHM_COD REC_ACDT

Das könnte Ihnen auch gefallen