Insurance Client S2 DE082 ETL Technical Specification v14

Solvency II
213560877.doc
Insurance Client Logo
INSURANCE CLIENT Solvency II

ETL Technical Specification
For Increment 2
Deliverable Reference
DE082
Version
1.14
Status
Signed-Off
File Name
213560877.doc
Last Edit Date
18/04/2012
Page 1 of 62
Solvency II
213560877.doc
Document Review
Name
Title
Date
Hitesh Lad
CMI ETL Lead
Randhir Vayalambrone
Solution Architect
Mark Carter
Application Delivery Manager
Sue Forder
IBM Solution Architect
Document Approval
Name
Title
Signature
Mark Carter
Application Delivery Manager
Version History
Author(s)
Version
Adrian Nagy
CR Number
Amendments/Reason
Date
1.0
Initial Publication
09/11/2011
Adrian Nagy
1.1
Updated with the SAP E2E

Reconciliation
10/11/2011
Adrian Nagy
1.2
Updated based on review

comments
15/11/2011
Adrian Nagy
1.3
Updated with the review

comments from Mark Carter
16/11/2011
Adrian Nagy
1.4
Added the reconciliations for

SOURCE 1-SAP and SOURCE
1-TDS; Updated the Section on
general ETL
18/11/2011
Adrian Nagy
1.5

comments from Paul Thomson,
Stuart Haynes, Sarah Wagstaff
and the IBM offshore team.
Updated with the changes from
the CRPR231 - SCS Process
and Hierarchy Changes.
The job names were updated.
30/11/2011
Adrian Nagy
1.6
The sections 3.4 and 3.5 were

updated to aggregate to the
UpperRID level instead of the
RID level.
The sections 3.7 and 4.3 were
updated based on the feedback
from the off-shore team.
01/12/2011
Adrian Nagy
1.7
Updated the E2E Reconciliation

as per feedback from the offshore team
05/12/2011
Adrian Nagy
1.8

comments from Hitesh Lad;
Updated with feedback from the
off-shore team
07/12/2011
Page 2 of 62
Date
Solvency II
213560877.doc
Adrian Nagy
1.9
Section 7.2 was updated to list

12/12/2011
the jobs which are populating the
metrics table
Adrian Nagy
1.10
Get_SCS_for_SOURCE 2
updated as per defect #282
Adrian Nagy
1.11
Updated for defect 319

Updated as per CR251
Changed the reconciliations
notifications as per defect 389
31/01/2012
Adrian Nagy
1.12
Updated as per defect 424;

Section 3.7 updated to not
compare the count on SOURCE
1-SAP reconciliation
Section 4.6.3 updated to refer to
the correct filename
06/02/2012
Adrian Nagy
1.13
Load Cover Snapshot was

updated for the partitioning
changes defect #440/#441
26/03/2012
Adrian Nagy
1.14
SAP Reference data metrics

removed defect #616
18/04/2012
Reference Documents
Ref
Number
Document
Version
Location of document
RD1
SII_R1.0_3.0_FD300 Single
Integration Layer_v1.10.docx
1.10
QuickR
RD2
SII_R1.0_2.9_FD300
SCS_Mapping_Functional_Desi
gn_document_1.0.docx
1.0
QuickR
RD3
INSURANCE
CLIENT_S2_TDS_V28.erwin
28
QuickR
RD4
Dist_and_risks v0.3.pptx
0.3
Email from Mark Carter
RD5
SCS tables v02.xls
0.2
Email from Leon van Heerden
RD6
FTP Order for Unitech Files
n/a
Email from Mark Carter
RD7
INSURANCE CLIENT S2 DE05

ETL Framework v1.1.pdf
1.1
QuickR
RD8
INSURANCE CLIENT S2
DE06b Source to Target
Mappings v0.1.xls
0.1
QuickR
RD9
Metric_codes_for_Reporting.xls
QuickR
Page 3 of 62
Solvency II
213560877.doc
TABLE OF CONTENTS
1 INTRODUCTION..................................................................................................................................... 7
1.1 Purpose............................................................................................................................................. 7
1.2 Scope for Increment 2....................................................................................................................... 7
1.3 SIL Tables......................................................................................................................................... 8
1.4 External Dependencies..................................................................................................................... 8
2 INCREMENT 2 ETL PROCESSING COMPONENT OVERVIEW...........................................................9
2.1 SIL Overview Diagram...................................................................................................................... 9
2.2 TDS to TDS Overview Diagram........................................................................................................ 9
2.3 Job Summary Table........................................................................................................................ 10
3 INTERNAL RECONCILIATION BUILD................................................................................................. 11
3.1 Jobs covered in this section............................................................................................................ 11
3.2 Structure and Mappings.................................................................................................................. 11
3.2.1 Data Input..................................................................................................................................... 11
3.2.2 Mapping....................................................................................................................................... 12
3.2.3 Data Output.................................................................................................................................. 12
3.3 SE022 SOURCE 1-TDS Reconciliation........................................................................................ 12
3.3.1 Description................................................................................................................................... 12
3.3.2 Job Flow....................................................................................................................................... 13
3.3.3 Build Specification........................................................................................................................ 13
3.4 SIL002SOURCE 1_TDS_Aggregate_and_compare.......................................................................14
3.4.1 Description................................................................................................................................... 14
3.4.2 Job Flow....................................................................................................................................... 14
3.5 SIL001ReadSOURCE 1Incoming ................................................................................................... 16
3.5.1 Description................................................................................................................................... 16
3.5.2 Job Flow....................................................................................................................................... 16
3.6 SE023 SOURCE 1-SAP Reconciliation........................................................................................ 17
3.6.1 Description................................................................................................................................... 17
3.6.2 Job Flow....................................................................................................................................... 17
3.7 SIL003SOURCE 1_SAP_Aggregate_and_compare.......................................................................18
3.7.1 Description................................................................................................................................... 18
3.7.2 Job Flow....................................................................................................................................... 18
4 SOURCE 1 SAP E2E RECONCILIATION BUILD..............................................................................20
4.2.1 Data Input..................................................................................................................................... 20
4.2.2 Mapping....................................................................................................................................... 21
4.2.3 Data Output.................................................................................................................................. 21
4.3 SE021 SOURCE 1-SAP E2E Reconciliation................................................................................21
4.3.1 Description................................................................................................................................... 21
4.3.2 Job Flow....................................................................................................................................... 22
4.4 SIL004_Load_SAP_Postings.......................................................................................................... 23
4.4.1 Description................................................................................................................................... 23
4.4.2 Job Flow....................................................................................................................................... 24
Page 4 of 62
Solvency II
213560877.doc
4.5 SIL005_Load_SAP_Errors.............................................................................................................. 24
4.5.1 Description................................................................................................................................... 24
4.5.2 Job Flow....................................................................................................................................... 24
4.6 SIL006_Load_SAP_Journal_and_Detail_Errors.............................................................................25
4.6.1 Description................................................................................................................................... 25
4.6.2 Job Flow....................................................................................................................................... 25
4.7 SIL007_E2E_Aggregate_and_compare.......................................................................................... 26
4.7.1 Description................................................................................................................................... 26
4.7.2 Job Flow....................................................................................................................................... 27
5 TDS TO TDS SNAPSHOT COMPONENT BUILD................................................................................30
5.2.1 Data Input..................................................................................................................................... 30
5.2.2 Mapping....................................................................................................................................... 31
5.2.3 Data Output.................................................................................................................................. 31
5.3 SE018 TDS Cover Snapshot for SOURCE 1...............................................................................31
5.3.1 Description................................................................................................................................... 31
5.4 SE019 TDS Cover Snapshot for SOURCE 2...............................................................................32
5.4.1 Description................................................................................................................................... 32
5.5 SE020 TDS Daily Measures......................................................................................................... 33
5.5.1 Description................................................................................................................................... 33
5.6 TDS001_Get_SCS_for_SOURCE 1................................................................................................ 33
5.6.1 Description................................................................................................................................... 33
5.6.2 Job Flow....................................................................................................................................... 33
5.7 TDS002_Get_SCS_for_SOURCE 2................................................................................................ 36
5.7.1 Description................................................................................................................................... 36
5.7.2 Job Flow....................................................................................................................................... 36
5.8 CTR001 Shared Container for Class Structure lookups...............................................................40
5.8.1 Description................................................................................................................................... 40
5.8.2 Container Flow............................................................................................................................. 41
5.9 TDS003_Load_Cover_Snapshot .................................................................................................... 42
5.9.1 Description................................................................................................................................... 42
5.9.2 Job Flow....................................................................................................................................... 42
5.10 TDS004_Calculate_Daily_Measures............................................................................................ 43
5.10.1 Description................................................................................................................................ 43
5.10.2 Job Flow.................................................................................................................................... 43
5.10.3 Build Specification..................................................................................................................... 43
5.11 TDS005_Load_Daily_Measures................................................................................................... 44
5.11.1 Description................................................................................................................................ 44
5.11.2 Job Flow.................................................................................................................................... 44
5.11.3 Build Specification..................................................................................................................... 44
6 POLICY HEADER - UPDATE TO INCREMENT 1 SIL TO TDS JOB...................................................46
Page 5 of 62
Solvency II
213560877.doc
6.2.1 Data Input..................................................................................................................................... 46

6.2.2 Mapping....................................................................................................................................... 46
6.2.3 Data Output.................................................................................................................................. 47
6.3 SE017 Load Transaction Tables.................................................................................................. 47
6.3.1 Description.................................................................................................................................. 47
6.3.2 Job Flow...................................................................................................................................... 47
6.3.3 Build Specification....................................................................................................................... 47
7 GENERAL ETL PROCESSING............................................................................................................. 49
7.1 Scheduling...................................................................................................................................... 49
7.1.1 E2E Reconciliation Batch Flow.................................................................................................... 49
7.1.2 SIL Batch Flow............................................................................................................................. 50
7.1.3 TDS Batch Flow........................................................................................................................... 51
7.1.4 Batch Implementation Details....................................................................................................... 51
7.2 Job Recording................................................................................................................................. 52
7.3 Error Management.......................................................................................................................... 57
7.4 Maintaining reference data.............................................................................................................. 59
7.5 Parameter Management.................................................................................................................. 59
7.6 File System Structure...................................................................................................................... 61
7.7 FTP File locations for Dev and Test................................................................................................ 62
Page 6 of 62
Solvency II
213560877.doc
1 Introduction
1.1 Purpose
The purpose of this document is to provide the detailed level technical specification for the ETL
components in scope of Increment 2 of the INSURANCE CLIENT Solvency II Release 0.1 solution. This
includes parts of the Single Integration Layer (SIL) and the load to the Snapshot and Measures Tables in
the Transactional Data Store (TDS). The document will describe the flow of the ETL processes which will
be used by the build team when developing ETL jobs.
1.2 Scope for Increment 2

The diagram below is an overview of all the ETL components within scope of Release 0.1 and those within
the red boxes are in scope of increment 2.
The key functional areas are:
Internal Reconciliation and Integrity Checking
SOURCE 1 SAP E2E Reconciliation
Classification followed by TDS Load for SOURCE 1 and SOURCE 2 to Cover Snapshot table
Calculations followed by TDS Load for SOURCE 1 and SOURCE 2 to Daily Measures
Page 7 of 62
Solvency II
213560877.doc
1.3 SIL Tables

There are a number of SQL Server tables that are used by the ETL jobs during the processing. This
schema is being maintained separately from the main TDS schema and will also be the subject of version
control.
1.4 External Dependencies

The following external dependencies have been identified:
DataStage Server Availability
Database Availability
Source File Availability
Page 8 of 62
Solvency II
213560877.doc
2 Increment 2 ETL Processing Component

Overview
2.1 SIL Overview Diagram
The diagram below shows the technical scope for the SIL for Increment 2 broken down into numbered jobs.
All ETL jobs within the solution will be prefixed with SE (Solvency II ETL). There is one job sequence for the
SOURCE 1 - SAP E2E reconciliation and two job sequences for the SOURCE 1-SAP and SOURCE 1-TDS
reconciliations.
INT 6
SE021
SAP
Reconciliation
Files
SAP E2E
Reconciliation
2.2 TDS to TDS Overview Diagram

The diagram below shows the technical scope for the TDS to TDS load for Increment 2 broken down into
numbered jobs.
Page 9 of 62
Solvency II
213560877.doc
SIFF to TDS
INT4
SIFF
Files
SE015
SE016
SE017
SIFF File Receipt
SIFF File
Validation
Load Transaction
Tables
Rejected SIFF
Files
Rejected SIFF
Records
SE018
SE020
TDS Cover
Snapshot for AIS
TDS Daily
Measures (AIS)
SE019
SE020
TDS Cover
Snapshot for
UKRIS
TDS Daily
Measures
(UKRIS)
Increment 2
2.3 Job Summary Table

Job Number (Name)
Description
SE018
Job Sequence used to get the SCS and load the

TDS_COVER_SNAPSHOT for SOURCE 1 data.
SE020
Job Sequence used to calculate the EP and UPR

measures and load the TDS_DAILY_MEASURES for
SOURCE 1 data.
SE019
Job Sequence used to get the SCS and load the

TDS_COVER_SNAPSHOT for SOURCE 2 data.
SE020
Job Sequence used to calculate the EP and UPR

measures and load the TDS_DAILY_MEASURES for
SOURCE 2 data.
SE021
Job Sequence used for the SOURCE 1 - SAP E2E

reconciliation.
SE022
Job Sequence used for the SOURCE 1 TDS

reconciliation
SE023
Job Sequence used for the SOURCE 1 SAP

reconciliation
SIL001ReadSOURCE 1Incoming
Job which writes the SOURCE 1 Incoming data into a

dataset.
Page 10 of 62
Solvency II
213560877.doc
3 Internal Reconciliation Build

3.1 Jobs covered in this section
The diagram below shows the job sequences and jobs used to reconcile data between SOURCE 1 and
SAP and between SOURCE 1 and TDS.
3.2 Structure and Mappings

3.2.1 Data Input
Input Name
Description
SIL_SOURCE 1_Journal_Incoming
Used to track progress of incoming SOURCE 1

journals through the mapping process and to hold as
errors.
SIL_SOURCE 1_Detail_Incoming
Used to track progress of incoming SOURCE 1 detail

records through the mapping process and to hold as
errors.
Page 11 of 62
Solvency II
213560877.doc
Input Name
Description
SIL_TDS_PT_Outgoing
Used to assemble a TDS_POLICY_TRANSACTION

record prior to transmitting the SIFnnnnn file.
SIL_TDS_PC_Outgoing
Used to assemble a
TDS_POLICY_CHARGES_INCOME record prior to
transmitting the SIFnnnnn file.
SIL_TDS_RC_Outgoing
Used to assemble a
TDS_POLICY_REINSURANCE_COVERAGE record
prior to transmitting the SIFnnnnn file.
SIL_SAP_Detail_Outgoing
Used to assemble a SAP Detail record prior to

transmitting the AIEnnnnnn file.
3.2.2 Mapping
Mapping
Description
N/A
3.2.3 Data Output

Output Name
Description
Reconciliation Success file
Will contain the results of the reconciliation in case of

success. Each row has Upper RID, count and sum
(amount)
Reconciliation Error file
Will contain the results of the reconciliation in case of

error. Each row has Upper RID, count and sum
(amount)
3.3 SE022 SOURCE 1-TDS Reconciliation

3.3.1 Description
This job sequence represents the overall flow needed to perform the SOURCE 1-TDS Reconciliation. The
components and activities in the sequence are detailed below in the build specification table.
Page 12 of 62
Solvency II
213560877.doc
3.3.2 Job Flow
3.3.3 Build Specification

DataStage
component
DataStage component type/name
Description
Sequence
name
SE022_SOURCE
1_TDS_RECONCILIATION
This job sequence executes the job which

aggregates the data from SOURCE 1 and
the data from SIL which was sent to TDS
and compares the results. It notifies the
persons responsible on the success or error
of the reconciliation process.
Job Activity
SIL002SOURCE
1_TDS_Aggregate_and_compare
Job which aggregates and compares

SOURCE 1 and TDS data.
Execute
Command
CMD
Checks if the reconciliation error file is

empty. If yes, the reconciliation is
successful.
Notification
Activities
Notifications
Notify the persons responsible on the

success or the failure of the reconciliation
process.
Include the reconciliation output file as an
attachment.
Parameters
Serial number
The sequence which is part of the filenames

for which the reconciliation process is done
DB Name
Database name
DB User
Database user name
DB Password
Database password
Email addresses
The email addresses of the people to be

notified in case of errors
Page 13 of 62
Solvency II
213560877.doc
3.4 SIL002SOURCE 1_TDS_Aggregate_and_compare

3.4.1 Description
This job aggregates the data from SOURCE 1 and the data sent to TDS and compares the results.
3.4.2 Job Flow

DataStage
component
Description
Job name
SIL002SOURCE
1_TDS_Aggregate_and_compare
This job aggregates and compares the data

from SOURCE 1 and data sent to TDS. It
reads and joins the SOURCE 1 data,
aggregates by grouping on the upper rid and
calculates the row count and the sum
(amount), does the same for the TDS data
and then compares the results. It writes a
success or an error file based on the result
of the comparison which contains the
reconciliation results.
Input Stage
ODBC Connector
Reads the Upper RID, count(*) and

sum(amount) from SIL_SOURCE
1_Detail_Incoming where
Derived_Amount_Type is blank.
Input Stage
ODBC Connector (SIL_TDS_PT_Outgoing)
Reads all records from the

SIL_TDS_PT_Outgoing table where the
Outgoing_Serial_No matches the serial
number for the current run (job parameter)
Input Stage
ODBC Connector
(SIL_TDS_PC_Outgoing)

SIL_TDS_PC_Outgoing table where the
Page 14 of 62
Solvency II
DataStage
component
213560877.doc
Description
Input Stage
ODBC Connector
(SIL_TDS_RC_Outgoing)

SIL_TDS_RC_Outgoing table where the
Input Stage
DataSet
The dataset containing the SOURCE 1

incoming data. The job which writes the
dataset is detailed on the next section.
Processing
Stage
Funnel
Unions the PC and RC records
Processing
Stage
Join
Joins the TDS data on the Policy ID
Processing
Stage
Column Generator
Stage which adds a column (rowcnt) to the

flow with the value 1. The aggregator will
then be used to sum on this column to
calculate the count of rows.
Processing
Stages
Aggregator
Aggregates by grouping on Upper RID and

calculating sum(rowcnt) and sum(amount)
Processing
Stage
Join
Joins the aggregation results from SOURCE

1 and TDS based on the Upper RID
Processing
Stage
Filter
Filters the reconciliation results by

comparing the counts and sums from
SOURCE 1 Incoming dataset minus the
counts and sums from the SOURCE 1
dropped records with TDS and redirects the
records to either success or error outputs.
Output Stages
Sequential Files
Two sequential files which will contain the

Parameters
Serial number

DB Name
Database name
DB User
Database user name
DB Password
Database password
Page 15 of 62
Solvency II
213560877.doc
3.5 SIL001ReadSOURCE 1Incoming

3.5.1 Description
This job joins and aggregates the data from SOURCE 1 and writes it to a dataset.
3.5.2 Job Flow

DataStage
component
Description
Job name
SIL001ReadSOURCE 1Incoming
This job joins and aggregates the data from

SOURCE 1 and writes it into a dataset
which can be used in the reconciliation
process.
Input Stage
ODBC Connector (SIL_SOURCE

1_Detail_Incoming)
Reads all records from the SIL_SOURCE

1_Detail_Incoming table where the
and both record_validated and
record_processed are true.
Processing
Stage
Column Generator
Stage which adds a column (rowcnt) to the

flow with the value 1. The aggregator will
Processing
Stage
Aggregator
Aggregates by grouping on Upper RID and

calculating sum(rowcnt) and sum(amount)
Output Stage
DataSet
Dataset containing the SOURCE 1 incoming

data.
Page 16 of 62
Solvency II
213560877.doc
DataStage
component
Description
Parameters
Serial number

DB Name
Database name
DB User
Database user name
DB Password
Database password
3.6 SE023 SOURCE 1-SAP Reconciliation

3.6.1 Description
This job sequence represents the overall flow needed to perform the SOURCE 1-SAP Reconciliation. The
components and activities in the sequence are detailed below in the build specification table.
3.6.2 Job Flow

DataStage
component
Description
Sequence
name
SE023_SOURCE
1_SAP_RECONCILIATION
This job sequence executes the job which

aggregates the data from SOURCE 1 and
the data from SIL which was sent to SAP
and compares the results. It notifies the
persons responsible on the success or error
of the reconciliation process.
Job Activity
SIL003SOURCE
1_SAP_Aggregate_and_compare

SOURCE 1 and SAP data.
Execute
CMD
Checks if the reconciliation error file is
Page 17 of 62
Solvency II
213560877.doc
DataStage
component
Command
Notification
Activities
Description
empty. If yes, the reconciliation is
successful.
Notifications

success or the failure of the reconciliation
process.
Include the reconciliation output file as an
attachment.
Parameters
Sequence number

DB Name
Database name
DB User
Database user name
DB Password
Database password
Email addresses

3.7 SIL003SOURCE 1_SAP_Aggregate_and_compare

3.7.1 Description
This job aggregates the data from SOURCE 1 and the data sent to SAP and compares the results.
3.7.2 Job Flow

=
Page 18 of 62
Solvency II
213560877.doc

DataStage
component
Description
Job name
SIL003SOURCE
1_SAP_Aggregate_and_compare

from SOURCE 1 and data sent to SAP. It
reads and joins the SOURCE 1 data,
aggregates by grouping on the upper rid and
calculates the sum (amount), does the same
for the SAP data and then compares the
results. It writes a success or an error file
based on the result of the comparison which
contains the reconciliation results.
Input Stage
ODBC Connector
(SIL_SAP_Detail_Outgoing)

SIL_SAP_Detail_Outgoing table.
Input Stage
DataSet
The dataset with the SOURCE 1 incoming

data. The dataset is being written by the
SIL001 job.
Processing
Stages
Aggregator
Aggregator stage which groups by Upper

RID and calculates sum(amount)
Processing
Stage
Join
Joins the aggregation results from SOURCE

1 and SAP based on the Upper RID
Processing
Stage
Filter
Filters the reconciliation results by

comparing the sums from SOURCE 1 with
SAP and redirects the records to either
success or error outputs.
Output Stages
Sequential Files
Two sequential files which will contain the

Parameters
Sequence number

DB Name
Database name
DB User
Database user name
DB Password
Database password
Page 19 of 62
Solvency II
213560877.doc
4 SOURCE 1 SAP E2E Reconciliation Build

The diagram below shows the Job Sequences and the Jobs used to reconcile the data posted by SAP with
the data from SOURCE 1 sent to TDS and SAP.
INT 6
SAP
Reconciliation
Files
SIL004
Load SAP
Postings
SIL005
SIL006
Load SAP Errors
Load SAP
Validation Errors
(Journal and
Detail)
SIL007
Aggregate and
Compare
SE021 SAP E2E Reconciliation

4.2.1 Data Input
Input Name
Description
aiennnnnnsap.dat
File containing SAP postings from the SAP

reconciliation data. The file format is documented in
the functional design document referenced as RD1.
aiennnnnnerr.dat
File containing SAP errors from the SAP

reconciliation data. The file format is documented in
the functional design document referenced as RD1.
aiennnnnne01
File containing SAP rejected journal and detail

records from the SAP reconciliation data. The file
format is documented in the functional design
document referenced as RD1.
Page 20 of 62
Solvency II
213560877.doc
4.2.2 Mapping
Mapping
Description
Source files to SIL_SAP tables
Simple one-to-one mapping from the files to the

tables. The mappings can be found in the Source to
Target Mapping spreadsheet that accompanies this
document (RD8).
4.2.3 Data Output

Output Name
Description
SIL_SAP_POSTING_INCOMING
Table used to capture SAP postings from the SAP

reconciliation data. The table structure is documented
in the functional design document referenced as
RD1.
SIL_SAP_ERRORS_INCOMING
Table used to capture SAP errors from the SAP

reconciliation data. The table structure is documented
in the functional design document referenced as
RD1.
SIL_SAP_JOURNAL_VALIDATION_ERRORS
Table used to capture SAP rejected journal records

from the SAP reconciliation data. The table structure
is documented in the functional design document
referenced as RD1.
SIL_SAP_DETAIL_VALIDATION_ERRORS
Table used to capture SAP rejected detail records

from the SAP reconciliation data. The table structure
is documented in the functional design document
referenced as RD1.
4.3 SE021 SOURCE 1-SAP E2E Reconciliation

4.3.1 Description
This job sequence represents the overall flow needed to perform the SOURCE 1-SAP E2E Reconciliation.
The components and activities in the sequence are detailed below in the build specification table.
Page 21 of 62
Solvency II
213560877.doc
4.3.2 Job Flow

DataStage
component
Description
Sequence
name
SE021_SOURCE
1_SAP_E2E_RECONCILIATION
This job sequence waits for the SAP files

to appear and then triggers the
reconciliation process. It gets the serial
number contained within the filename,
deletes the data from the SIL SAP tables
for this serial number (if the file has been
loaded previously), loads the data into
SIL SAP tables, aggregates the data
from SAP and the data from SIL and
compares the results. If the reconciliation
is successful, deletes the data from the
SAP tables. If the reconciliation is not
successful, notifies the persons
responsible for future investigation.
The sequence will be scheduled to run
multiple times in a batch as there may be
more than a set of files needed to
reconcile and they will be processed one
by one.
Wait Activity
Wait for file
This activity will wait for the file

aiennnnnnerr.dat to appear before it
triggers the reconciliation process. This
file will be pushed last, so it is certain
that the others were delivered when this
file appears.
Execute
Command
CMD
Script which waits for the error file to be

unlocked (the write process is finished)
and gets the sequence part of the
filename and writes it to a variable to be
used further in the flow. It should sort the
files on that location based on the serial
number (there maybe more than one
serial number present) and take the first
set.
Page 22 of 62
Solvency II
213560877.doc
DataStage
component
Description
Execute
Command
CMD
Script which deletes the records with the

same filename from the SAP tables in
SIL.
Job Activity
SIL004_Load_SAP_Postings
Job which loads the SAP Postings.
Job Activity
SIL005_Load_SAP_Errors
Job which loads the SAP Errors.
Job Activity
SIL006_Load_SAP_Journal_and_Detail_Errors
Job which loads the SAP Journal and

Detail Errors.
Job Activity
SIL007_E2E_Aggregate_and_compare

SAP and SOURCE 1 SIL data.
Execute
Command
CMD
Checks if the reconciliation is successful.
Execute
Command
CMD
If the reconciliation is successful,

archives and then deletes both the
source files and the records from the
SAP tables.
Notification
Activities
Notifications

success or the failure of the
reconciliation process.
Include the reconciliation output file as
an attachment.
Parameters
Source File Location
The path to the source files
DB Name
Database name
DB User
Database user name
DB Password
Database password
Email addresses

4.4 SIL004_Load_SAP_Postings
4.4.1 Description
This job loads the SIL_SAP_POSTINGS_INCOMING table in SIL with the reconciliation data.
Page 23 of 62
Solvency II
213560877.doc
4.4.2 Job Flow

DataStage
component
Description
Job name
This job loads the aiennnnnnsap.dat data

into the corresponding SIL table.
Input Stage
Sequential File
Reads the source file to be processed

within the current batch.
Processing
Stage
Copy stage
Copies the input to the output, with oneto-one mapping for all the fields.
Output Stage
ODBC Connector Stage
Loads the
SIL_SAP_POSTINGS_INCOMING table
in SIL.
Parameters
Source filename sequence
Part of the filename used to identify the

files
Database name
DB Name
Database user name
DB User
Database password
DB Password
4.5 SIL005_Load_SAP_Errors
4.5.1 Description
This job loads the SIL_SAP_ERRORS_INCOMING table in SIL with the reconciliation data.
4.5.2 Job Flow
Page 24 of 62
Solvency II
213560877.doc

DataStage
component
Description
Job name
This job loads the aiennnnnnerr.dat data

into the corresponding SIL table.
Input Stage
Sequential File

Processing
Stage
Copy stage
Copies the input to the output
Output Stage
Loads the
SIL_SAP_ERRORS_INCOMING table in
SIL.
Parameters

files
Database name
DB Name
Database user name
DB User
Database password
DB Password
4.6 SIL006_Load_SAP_Journal_and_Detail_Errors
4.6.1 Description
This job loads the SIL_SAP_ JOURNAL_VALIDATION_ERRORS and
SIL_SAP_DETAIL_VALIDATION_ERRORS tables in SIL with the reconciliation data.
4.6.2 Job Flow
Page 25 of 62
Solvency II
213560877.doc

DataStage
component
Description
Job name
This job loads the journal and the detail

validation errors data
(aiennnnnne01.dat) into the
corresponding SIL tables.
Input Stage
Sequential File

Processing
Stage
Modify Stage
Performs substring of the first two

characters of the input and place the
values into a new column
Processing
Stage
Filter stage
Checks the values in the new column

added in the modify stage and captures
the records as follows:
05 = Journal Record
10 = Detail Record
Output Stage
Loads the SIL_SAP_

JOURNAL_VALIDATION_ERRORS
table in SIL.
Output Stage
Loads the SIL_SAP_

DETAIL_VALIDATION_ERRORS table
in SIL.
Parameters

files
Database name
DB Name
Database user name
DB User
Database password
DB Password
4.7 SIL007_E2E_Aggregate_and_compare
4.7.1 Description
This job performs the aggregation of the SAP and SIL data and compares the results.
Page 26 of 62
Solvency II
213560877.doc
4.7.2 Job Flow

DataStage
component
Description
Job name
SIL007_E2E_Aggregate_and_compare

from SAP and SOURCE 1-SIL. It reads and
joins the SAP data, aggregates on the serial
number and calculates the row count and the
sum(amount), does the same for the SIL data
and then compares the results. It writes a
success or an error flag based on the result
of the comparison.
Input Stage
This stage reads the data from the

SIL_SAP_Postings_Incoming table filtered by
the posted_serial_number equals the serial
number parameter.
Input Stage

SIL_SAP_Errors_Incoming table filtered by the
posted_serial_number equals the serial
number parameter.
Input Stage

SIL_SAP_Detail_Validation_Errors_Incoming
table joined with
SIL_SAP_Journal_Validation_Errors_Incoming
on Journal_Id and filtered by the
posted_serial_number equals the serial
number parameter.
Input Stage
This stage reads the SIL SAP data from the

SIL_SOURCE 1_Detail_Incoming table. It
Page 27 of 62
Solvency II
DataStage
component
213560877.doc
Description
filters the records on the serial number equals
outgoing_serial_number and
record_processed=true (the data which was
sent to SAP).
Processing
Staged
Copy stages
Three copy stages used to rename the source

columns to bring them to the same names for
the funnel stage. Keep only the Serial no and
Amount columns.
Processing
Stage
Funnel stage
Funnel stage to union the SAP Postings and

Errors data.
Processing
Stage
Aggregator Stage
Aggregator stage which will group on the

below keys and will sum(amount):
- RecordValidated
- Company_Code
- Currency_Code
- Posting_Date
- SAP_Entry_Number
- Policy_Number
- Policy_Version
- Treasury_Job_Number
- Receivable_Id
- Job_Date
- Posting_Key_Debit
- Posting_Key_Credit
This is performed to bring the SIL data to the
same level of detail as the SAP data.
Processing
stage
Join Stage
Join stage on the serial number. Inner join.
Processing
Stages
Column Generators
Two stages which add a column (rowcnt) to

the flow with the value 1. The aggregator will
Processing
stage
Aggregator Stage
Aggregator stage to make the sum(rowcnt)

and the sum(amount). The grouping is done
on the serial number.
Processing
stage
Filter Stage
Compares the count and the sum calculated in

the aggregator stages. If both matches go to
the success output link; if not, go to the error
Page 28 of 62
Solvency II
DataStage
component
213560877.doc
Description
output link.
Output Stages
Sequential File
A sequential file which will contain 0/1 on the

first line to indicate the reconciliation was
successful or not. It will also contain the
reconciliation figures (both count and sum of
amount). Will be read from inside the job
sequence.
Parameters
File Location
The path where the output file is to be written
Source filename serial number
Part of the filename used to identify the files
DB Name
Database name
DB User
Database user name
DB Password
Database password
Page 29 of 62
Solvency II
213560877.doc
5 TDS to TDS Snapshot Component Build

The diagram below shows the Job Sequences and the Jobs used to populate the TDS Cover Snapshot and
the TDS Daily Measures tables. For TDS Cover Snapshot, there are separate job sequences for each
source systems: SOURCE 1 and SOURCE 2 but they are calling the same jobs, except for the
determination of the SCS where the logic and the fields used are completely different. For the TDS Daily
Measures there is only one job sequence. The jobs and job sequences which are shared by SOURCE 1
and SOURCE 2 will have a parameter to specify the Source System and the value of this parameter should
be part of the dataset filename where appropriate.

5.2.1 Data Input
Input Name
Description
TDS Policy Transaction Table
The primary fact table holding transaction level
Page 30 of 62
Solvency II
213560877.doc
Input Name
Description
premium data.
5.2.2 Mapping
Mapping
Description
TDS Policy Transaction to TDS Cover Snapshot

and TDS Daily Measures
Mapping that details how the fields in the source

tables are mapped into the TDS Cover Snapshot and
Daily Measures Tables.
This can be found in the Source to Target Mapping
spreadsheet that accompanies this document (RD8).
5.2.3 Data Output

Output Name
Description
TDS Cover Snapshot
This table holds cover data at the lowest available

level together with the SCS code appropriate to the
policy. Policy ID allows linking back up to the Parent
policy of which this cover is a part.
TDS Daily Measures
This table holds Unearned Premium Reserve (UPR)

and Earned Premium (EP) data for each individual
cover. It is recalculated on a scheduled basis (either
daily or monthly) and the date of calculation is
recorded. Measures are valid only on that date. The
daily change in EP and UPR is held on this table to
facilitate ad-hoc calculation.
5.3 SE018 TDS Cover Snapshot for SOURCE 1

5.3.1 Description
This job sequence executes the jobs which populate the TDS Cover Snapshot for the SOURCE 1 source
system.

DataStage
component
Description
Job Sequence
name
SE018_TDS_Cover_Snapshot_SOURCE
1
This sequence executes the jobs which

populate the TDS Cover Snapshot for the
Page 31 of 62
Solvency II
213560877.doc
DataStage
component
Description
SOURCE 1 source system.
Job Activity
TDS001_Get_SCS_for_SOURCE 1
The job which gets the SCS for the SOURCE

1 data.
Job Activity
TDS003_Load_Snapshot_for_SOURCE 1
The job which loads the TDS Cover Snapshot

table. The value of the Source system
parameter must be specified as SOURCE 1.
Parameters
Source System
Specifies the Source System (either

SOURCE 1 or SOURCE 2) the data is to be
loaded from. Used in the load job.
5.4 SE019 TDS Cover Snapshot for SOURCE 2

5.4.1 Description
This job sequence executes the jobs which populate the TDS Cover Snapshot for the SOURCE 2 source
system.

DataStage
component
Description
Sequence
name
SE019_TDS_Snapshot_SOURCE 2

populate the TDS Cover Snapshot for the
SOURCE 2 source system.
Job Activity
The job which gets SCS for SOURCE 2 data
Job Activity
TDS003_Load_Snapshot_for_SOURCE 2
The job which loads the TDS Cover Snapshot

table. The value of the Source system
parameter must be specified as SOURCE 2.
Parameters
Source System

Page 32 of 62
Solvency II
213560877.doc
5.5 SE020 TDS Daily Measures

5.5.1 Description
This job sequence executes the jobs which populate the TDS Daily Measures for the SOURCE 1 and
SOURCE 2 source systems. The job sequence has a parameter to specify the Source System and its value
it is passed to the executed jobs. The transformation logic does not depend on the source hence there is no
need to build separate jobs or job sequences for each source system as in the TDS Cover Snapshot case.

DataStage
component
Description
Sequence
name
SE020_TDS_Daily_Measures

populate the TDS Daily Measures for the
SOURCE 1 and SOURCE 2 source system.
Job Activity
TDS004_Calculate_Daily_Measures
The job which calculates the daily measures.
Job Activity
TDS005_Load_Daily_Measures
The job which loads the TDS Daily Measures

table.
Parameters
Source System

5.6 TDS001_Get_SCS_for_SOURCE 1
5.6.1 Description
This job will map the SOURCE 1 Policy Transaction data to SCS.
5.6.2 Job Flow
Page 33 of 62
Solvency II
213560877.doc

DataStage
component
Description
Job name
This job gets the Distribution

Channel and the Risk Class and
then using these it gets the SCS ID
and the other IDs in the Class
Structure Map. It generates a
surrogate key which is to be used
as a primary key for the snapshot
table and writes the data into a
dataset using Overwrite method.
Input Stage

(DB_TDS_POLICY_TRANSACTION_SOUR
CE 1)

TDS_POLICY_TRANSACTION
table filtered by the source system
equals SOURCE 1. This value is
specified in a parameter. Also filter
only new transactions since the last
load by reading the rows with the
policy_transaction_id higher than
the max_policy_transaction_id
which is to be specified as a
parameter.
Input Stage

(DB_SCS_DISTRIBUTION_CHANNEL_RUL
ES)
This stage reads the Distribution

Channel Rules table. Select only
the columns to be used in the
lookup.
Input Stage

(DB_SCS_RISK_CLASS)
This stage reads the Risk Class

Rules table. Select only the
columns to be used in the lookup.
Processing Stage
Lookup
(LKP_DISTRIBUTION_CHANNEL_RULES)
Lookup stage to get the Channel

Code. Lookup should continue
when not found and the Channel
Code should be made nullable.
The lookup condition is described in
the source-to-target mappings. Add
the range condition for the date
(snapshot date between start date
and end date) to make sure that the
lookup returns the latest record.
Change the partitioning method on
the reference link to Entire.
Processing Stage
Lookup (LKP_RISK_CLASS_RULES)
Lookup stage to get the Risk Code.

Lookup should continue when not
found and the Risk Code should be
made nullable. The lookup
condition is described in the source-
Page 34 of 62
Solvency II
DataStage
component
213560877.doc
Description
to-target mappings. Add the range
condition for the date (snapshot
date between start date and end
date) to make sure that the lookup
returns the latest record. Change
the partitioning method on the
reference link to Entire.
Container
Shared Container
(CTR_GET_CLASS_STRUCTURE)
Embed the named shared

container. Match the input and the
output links. The specifications of
the shared container can be found
in the section 5.8 below.
Processing Stage
Copy Stage (CP_TO_TGT_DS)
Copy stage to drop/rename

columns as appropriate. Even if no
column will be dropped or renamed;
it is a good practice to use a copy
stage before the target stage as a
placeholder to make the future
changes easier to be implemented.
Output Stage
Dataset File
(DS_TDS_COVER_SNAPSHOT)
Dataset which holds the processed

records to be written in the target
table TDS_COVER_SNAPSHOT.
The dataset name will contain
SOURCE 1 to indicate that these
are SOURCE 1 records. The
dataset will be picked up as source
by the load job.
Parameters
Dataset Location
The path where the datasets are

stored.
Database name
DB Name
Database User Name
DB USER
Database Password
DB PWD
=SOURCE 1
Source System
The date the snapshot is created

for.
Snapshot Date
Max policy transaction id
Page 35 of 62
The maximum value of the policy

transaction id before the current
batch run. This is to be updated in
the parameter set file from the
overall job sequence, after each run
of the cover snapshot for all the
source systems.
Solvency II
213560877.doc
5.7 TDS002_Get_SCS_for_SOURCE 2
5.7.1 Description
This job will map the SOURCE 2 Policy Transaction data to SCS.
5.7.2 Job Flow

DataStage
component
Description
Job name
This job gets the Distribution

Channel and the Risk Class and
then using these it gets the SCS ID
and the other IDs in the Class
Structure Map. It generates a
surrogate key which is to be used
as a primary key for the snapshot
table and writes the data into a
dataset using Overwrite method.
Input Stage

(DB_TDS_POLICY_TRANSACTION_SOUR
CE 2)

TDS_POLICY_TRANSACTION
table filtered by the source system
Page 36 of 62
Solvency II
DataStage
component
213560877.doc
Description
equals SOURCE 2. This value is
specified in a parameter. Also filter
only new transactions since the last
load by reading the rows with the
policy_transaction_id higher than
the max_policy_transaction_id
which is to be specified as a
parameter.
Input Stage

(DB_SCS_DISTRIBUTION_CHANNEL_RUL
ES)
This stage reads the Distribution

Channel Rules table. Select only
the columns to be used in the
lookup.
Input Stage

(DB_SCS_RISK_CLASS_RULES)
This stage reads the Risk Class

Rules table. Select only the
columns to be used in the lookup.
Processing Stage
Transformer Stage (CP_FIELD_MAPPINGS)
Use a Transformer stage to send

the reference data to three outputs
which are reference links for the
three lookups. Send the appropriate
columns for each lookup as
described in the source to target
mappings.
For the third lookup, send only the
data with Agency_Id=0
Processing Stage
Lookup (LKP_DIST_CHANNEL_RULES_1)
First lookup stage to get the

Distribution Channel ID. Lookup
should reject when not found and
the reject link should be the stream
for the next lookup. The lookup
condition is described in the sourceto-target mappings. Add the range
Processing Stage
Lookup (LKP_DIST_CHANNEL_RULES_2)
Second lookup stage to get the

Distribution Channel ID. Lookup
should reject when not found and
the reject link should be the stream
for the next lookup. The lookup
condition is described in the sourceto-target mappings. Add the range
Page 37 of 62
Solvency II
DataStage
component
213560877.doc
Description
Processing Stage
Lookup
(LKP_DISTRIBUTION_CHANNEL_RULES)
Third lookup stage to get the

Channel Code. Lookup should
continue when not found and the
Channel Code should be made
nullable. The lookup condition is
described in the source-to-target
mappings. Add the range condition
for the date (snapshot date
between start date and end date) to
make sure that the lookup returns
the latest record. Change the
partitioning method on the
Processing Stage
Funnel (FNL_DISTRIBUTION_CHANNEL)
Funnel stage used to union all the

rows.
Processing Stage
Transformer (CP_FIELD_MAPPINGS_RISK)
Use a transformer stage to send the

reference data to three outputs
which are reference links for the
three lookups. Send the appropriate
columns for each lookup as
described in the source to target
mappings.
For the second lookup, send only
the data with Policy_Number=0
For the third lookup, send only the
data with Policy_Number=0 and
Coverage_Id=0
Processing Stage
Lookup (LKP_RISK_CLASS_RULES_1)
First Lookup stage to get the Risk

Class ID. Lookup should have a
reject link to go to the next lookup.
The lookup condition is described in
the source-to-target mappings. Use
the priority column plus the key
fields to sort on the reference data
to ensure that the highest priority
record id is matched.
Add the range condition for the date
Page 38 of 62
Solvency II
213560877.doc
DataStage
component
Description
Processing Stage
Second Lookup stage to get the

Risk Class ID. Lookup should have
a reject link to go to the next
lookup. The lookup condition is
described in the source-to-target
mappings. Use the priority column
plus the key fields to sort on the
reference data to ensure that the
highest priority record id is
matched.
Processing Stage
Third lookup stage to get the Risk

Class ID. Lookup should continue
when not found and the Risk Code
should be made nullable. The
lookup condition is described in the
source-to-target mappings. Use the
priority column plus the key fields to
sort on the reference data to ensure
that the highest priority record id is
matched.
Processing Stage
Funnel (FNL_RISK)
Funnel stage used to union all the

rows.
Container
Shared Container
(CTR_GET_CLASS_STRUCTURE)
Embed the named shared

container. Match the input and the
output links. The specifications of
the shared container can be found
in the section 5.8 below.
Processing Stage
Copy Stage (CP_TO_TGT_DS)
Copy stage to drop/rename

columns as appropriate. Even no
column will be dropped or renamed;
it is a good practice to use a copy
stage before the target stage as a
placeholder to make the future
changes easier to be implemented.
Page 39 of 62
Solvency II
213560877.doc
DataStage
component
Description
Output Stage
Dataset File
(DS_TDS_COVER_SNAPSHOT)

records to be written in the target
table TDS_COVER_SNAPSHOT.
The dataset name will contain
SOURCE 2 to indicate that these
are SOURCE 2 records. The
dataset will be picked up as source
by the load job.
Parameters
Dataset Location
The path where the datasets are

stored.
Database name
DB Name
Database User Name
DB USER
Database Password
DB PWD
=SOURCE 2
Source System
The date the snapshot is created

for.
Snapshot Date
Max policy transaction id
The maximum value of the policy

transaction id before the current
batch run. This is to be updated in
the parameter set file from the
overall job sequence, after each run
of the cover snapshot for all the
source systems.
5.8 CTR001 Shared Container for Class Structure

lookups
5.8.1 Description
This shared container is used from within the Get SCS SOURCE 1 and SOURCE 2 jobs. It gets the Class
Structure IDS by looking up based on the Distribution Channel and Risk Class which should be on the input
link.
Page 40 of 62
Solvency II
213560877.doc
5.8.2 Container Flow

DataStage
component
Description
Input Stage

(DB_TDS_CLASS_STRUCTURE_MATRIX)
This stage reads the Class Structure

Matrix table. Select only the columns to
be used in the lookup.
Input Stage

(DB_TDS_CLASS_STRUCTURE_MAP)
This stage reads the Class Structure Map

table. Select only the columns to be used
in the lookup.
Processing
Stages
Modify Stages (MDY_HANDLE_NULL)
Three Modify stages to make a

null_to_value to transform all the null
values in the distribution channel and risk
class ids and then scs id and then HRG,
S2, GBM, IM and Planning IDs (records
not matched in the previous lookups) to
the default unmapped key and change
their nullability. The unmapped key to be
used is -1.
Processing
Stage
Lookup Stage
(LKP_CLASS_STRUCTURE_MATRIX)
Lookup stage to the SCS class structure

matrix table with the channel code and
risk code to get the scs_id and the
allocation, as described in the source to
target spreadsheet. Use Multiple rows
returned from link as different classes
might be returned for the same policy.
Processing
Stage
Lookup Stage
(LKP_CLASS_STRUCTURE_MAP)
Lookup SCS class structure map to get

HRG, S2, GBM, IM and Planning IDs.
Use the condition as specified in the
source to target mappings.
Processing
Stage
Surrogate Key Generator Stage
Surrogate key generator used to

generate the Cover Snapshot ID which is
the Primary Key of the Snapshot table.
Page 41 of 62
Solvency II
213560877.doc
5.9 TDS003_Load_Cover_Snapshot
5.9.1 Description
This job loads the TDS_COVER_SNAPSHOT table.
5.9.2 Job Flow

DataStage
component
Description
Job name
TDS003_Load_Cover_Snapshot
This job reads the DataSet created by

the Get SCS jobs and loads it into the
TDS_COVER_SNAPSHOT table.
Input Stage
Dataset (DS_TDS_COVER_SNAPSHOT)
The dataset written by the Get SCS jobs.

Use a parameter to specify the source
system.
Processing
Stage
Transformer
(XFM_TDS_COVER_SNAPSHOT)
Transformer stage, calculate the

measures as detailed in the mappings
spreadsheet.
Output Stage

(DB_TDS_COVER_SNAPSHOT)
Writes to the TDS_COVER_SNAPSHOT

table. Given the data volume we need a
way to manage the re-processing and reloading; the upsert method will not work
very well. Partition the table daily on the
snapshot date and manage the partitions
from within DataStage use a
parameterized stored procedure which
should be called as Before SQL on the
ODBC Stage. It checks if the data is
already loaded for the snapshot date and
if so, drop the corresponding partition
and create it again.
Parameters
Dataset Location
The path where the datasets are stored.
DB Name
Database name
DB USER
Database User Name
DB PWD
Database Password
Source System
The source system name.
Page 42 of 62
Solvency II
213560877.doc
5.10
5.10.1
Description
This job calculates all the Earned Premium and Unearned Premium Reserve daily measures.
5.10.2
Job Flow
5.10.3
Build Specification
DataStage
component
Description
Job Name
This job reads the snapshot data, uses a

transformer to calculate the measures,
generates a surrogate key and writes the
data into a dataset.
Input Stage

(DB_TDS_COVER_SNAPSHOT)
This reads the

TDS_COVER_SNAPSHOT table. Join
with the TDS_POLICY_HEADER on
Policy ID and use a parameter to specify
the Source System to filter the data on.
Also filter on the records with the
calculation date between effective date
and end date.
Processing
Stage
Transformer Stage
(XFM_CALCULATE_MEASURES)
Transformer stage for the calculation of

the UPR and EP (the formulas are
specified in the mappings). Use variables
inside the transformer to avoid
calculating the same value more than
once.
Processing
Stage
Surrogate Key Generator Stage
Surrogate key generator used to

generate the Daily Measures ID which is
the Primary Key of the Daily Measures
table.
Output Stage
Dataset File
(DS_TDS_DAILY_MEASURES)

records to be written in the target table
TDS_DAILY_MEASURES. The dataset
name will contain Source System as part
of the name to indicate which records are
stored in it. The dataset will be picked up
as source by the load job.
Page 43 of 62
Solvency II
213560877.doc
DataStage
component
Description
Parameters
Dataset Location
DB Name
Database name
DB USER
Database User Name
DB PWD
Database Password
Source System
Calculation Date
The date the calculations are made for.
5.11
5.11.1
Description
This job loads the TDS_DAILY_MEASURES table.
5.11.2
Job Flow
5.11.3
Build Specification
DataStage
component
Description
Job Name
This job reads the DataSet created by

the Calculate Measures job and loads the
data into the TDS_DAILY_MEASURES
table.
Input Stage
Dataset (DS_TDS_DAILY_MEASURES)
The dataset written by the Calculate the

Daily Measures job. Use a parameter to
specify the source system.
Processing
Stage
Copy Stage
(CP_TDS_DAILY_MEASURES)
Copy stage, no transformations or

mappings required.
Output Stage

(DB_TDS_DAILY_MEASURES)
Writes to the TDS_DAILY_MEASURES

table. Given the data volume we need a
way to manage the re-processing and reloading; the upsert method will not work
very well. Partition the table daily on
calculation date and manage the
partitions from within DataStage use a
Page 44 of 62
Solvency II
DataStage
component
213560877.doc
Description
parameterized stored procedure which
should be called as Before SQL on the
ODBC Stage. There are two actions
which need to be done:
Parameters
check if the data is already

loaded for the calculation date
and if so, based on the value of a
parameter, either drop the
partition and create it again or
simply add data to the same
calculation date
drop the partitions which are

older than 3 calendar months
plus one week, except the
partitions for the last day in each
month as we need to keep three
months rolling of history plus
end-month figures
Dataset Location
DB Name
Database name
DB USER
Database User Name
DB PWD
Database Password
Source System
Page 45 of 62
Solvency II
213560877.doc
6 Policy Header - Update to Increment 1 SIL to

TDS job
This section shows the update needed to be done to the increment 1 SE017 job to load the TDS Policy
Header table.
SE017
Load Transaction
Tables
Rejected SIFF
Records

6.2.1 Data Input
Input Name
Description
New input: TDS_Policy_Header
There is one new input to be added to the job which

is the Policy Header table to be used to lookup
against to get the Policy_ID.
6.2.2 Mapping
Mapping
Description
SIFF data to TDS Policy Header and TDS Policy

Transaction
Lookup the Policy Header with Source System ID,

Entity ID, Policy Number, Policy Number 2, Policy
Number 3, Policy Version Number, Policy Type ID
and Client ID to get the Policy ID.
Page 46 of 62
Solvency II
213560877.doc
6.2.3 Data Output

Output Name
Description
New output: TDS_Policy_Header
There is one new output to add to this job, the Policy

Header table. This table holds policy header level
data.
6.3 SE017 Load Transaction Tables

6.3.1 Description
This is the Increment 1 job which needs to be modifying to include the TDS Policy Header lookup and
population. The flow and the specification below refer only to the changes needed to be implemented.
This change needs to be implemented in the job called TDS004LoadPolicyTransactionDS rather than
TDS009Policy_TransactionLoadDB, as the FK constraint requires data to be loaded in the Policy Header
before it is loaded to the Policy Transaction.
6.3.2 Job Flow

DataStage
component
Description
Input Stage
ODBC Connector
Reads the TDS Policy Header data
Processing
Stage
Lookup
Lookup TDS Policy Header with the fields

specified in the mappings above to get
Page 47 of 62
Solvency II
DataStage
component
213560877.doc
Description
the Policy ID.
If the lookup fails, go to the reject link and
continue with the Surrogate Key
Generator stage.
If the lookup is successful, go to the
Funnel stage.
Processing
Stage
Surrogate Key Generator (Policy ID)
When the lookup to the Policy Header is

not successful, this stage generates a
new value to be used as a Policy ID.
Processing
Stage
Copy
Copies the input to two outputs. Drop the

unnecessary columns on each output.
Processing
Stage
Funnel
Unions the records
Output Stage
ODBC Connector
Writes to the TDS Policy Header table.
Page 48 of 62
Solvency II
213560877.doc
7 General ETL Processing

This section describes the General ETL processing elements that support the functional aspects of the
system described in the sections above.
7.1 Scheduling
This section shows the batch schedule and flow for both SIL & TDS. It then also provides details about how
this will be implemented within DataStage.
7.1.1 E2E Reconciliation Batch Flow
The diagram above shows the E2E Reconciliation Batch Flow. The process will be triggered by the
presence of the last file expected from SAP as defined in the corresponding section above. This process is
independent of the other flows.
Page 49 of 62
Solvency II
213560877.doc
7.1.2 SIL Batch Flow
The diagram above highlights the Increment 2 flows which will be placed inside the overall SIL batch.
Page 50 of 62
Solvency II
213560877.doc
7.1.3 TDS Batch Flow
The diagram above highlights the Increment 2 flows which will be placed in the overall TDS Batch.
7.1.4 Batch Implementation Details

The batch will be implemented as follows:
Time Based Sequences

These will be scheduled to run in the DataStage Scheduler
File Based Sequences

These will be scheduled to run in the DataStage Scheduler, however they will be controlled by file
watchers over a period of time. E.g. the SOURCE 1 sequence will be started at 3am by the
scheduler and then it will continue to watch for files until 7am.
Manual Triggering
Should there be a requirement to trigger a sequence manually this can be done from DataStage
Director.
Parallel & Multiple File Processing

The SOURCE 2 & SOURCE 1 sequences can run in parallel, however they will only process one
file at a time. Therefore, if multiple files are present for SOURCE 1 or SOURCE 2 they will be
processed sequentially to maintain the integrity of the RID allocation process.
Restartability
The sequences will be designed to be restartable, such that if there is a system failure they can be
restarted and will pick up from where they stopped. They will also have controls in place that check
whether the sequence can run e.g. are the intermediate tables clear, is a sequence already
running, is the sequence in a runnable state (i.e. not aborted).
Page 51 of 62
Solvency II
213560877.doc
7.2 Job Recording

Job recording is the component that logs information about the executing DataStage job such as start time,
duration, and volume of data processed and whether the job completed successfully or failed.
Each job in DataStage is identified by a unique ID and each execution of a job is identified by a unique
batch number. Batch numbers may be the same for more than one job, for example, a series of jobs that
make up a processing schedule.
The table below details the ETL Control Tables:
CTRL_JOB_DEFN
Column name
Data Type
Key
Description
Job_id
INT
PK
A unique ID for the Job
Job_name
Varchar(100)
A short name for the job to

identify the job in reports
Source_System_Name
Varchar(20)
SOURCE 1/SOURCE 2
Job_description
Varchar(200)
Description of the job.
CTRL_BATCH_EXEC
Column name
Data Type
Key
Description
JOB_EXECUTION_ID
INT
PK
A unique ID for the execution
Batch_id
INT
Job_id
INT
FK (CTRL_JOB_Defn)
Serial_No
Varchar(6)
Last 6 character of File name.

i.e.nnnnnn from the file name.
Business_Date
Date
(YYYYMMDD) The business date read

from the header of the source file
Job_Start_Date_Time
DateTime
Start Date and Time for the job
Job_End_Date_Time
DateTime
Completion Date and Time
User_Name
Varchar(20)
Status
Varchar2(20)
A unique ID for the batch
Success or failure for the job.

Values include: Completed
without error, completed with
errors logged, aborted, failed.
Where "aborted" is for jobs
Page 52 of 62
Solvency II
213560877.doc
that deliberately abort while

"failed" is where the job was
forced to terminate, for
example system failure.
CTRL_EXEC_METRIC
Column name
Data Type
Key
Description
Metric_ID
INT
PK
A unique ID for the metric

being recorded
Job_Execution_ID
INT
FK (CTRL_Batch_Exec) The ID for the execution

against which metrics are being recorded
Metric_Code
Varchar(5)
Metric code
Metric_Name
Varchar(255)
Name for the metric
Metric_Value
Varchar(100)
Value for the metric
Metrics in the CTRL_EXEC_METRIC table include:
Processed file name (fully qualified)
Process file number
Size of file (bytes)
Row count
Financial Total
Hash Sum Total
Output file name (fully qualified)
Output file size (bytes)
Output file row count
Output file financial Total
Reject file name (fully qualified)
Reject file size (bytes)
Reject file total row count
Reject file size
Count of records failing data validation
Count of records failing mapping
The Metric_Code is indicating the system, direction of the flow, type of records, the metric type and the
subject. Example: Total record Count Extracted from Source System (either SOURCE 1 or SOURCE 2) will
have the Metric_Code = SINR[A,U]. The detail description of the codes can be found in the RD9.
Page 53 of 62
Solvency II
213560877.doc
The following table shows the job sequences and the jobs which will record the information to the
CTRL_EXEC_METRIC table.
Job sequence name
Job name
SE003_Load_SOURCE
1_Receipt_Tables
SOURCE 1001ExtractSOURCE 1SourceFile
SE003_Load_SOURCE
1_Receipt_Tables
SOURCE 1001ExtractSOURCE 1SourceFile
Metrics recorded
Number of Read (New

Records):Records,Financ
Totals,Policies,Policy Ver
Number of Loaded to DB
JournalIncoming, DetailIn
(New Records):Records,F
Totals,Policies,Policy Ver
SE004_SOURCE 1_Data_Validation
SOURCE 1001DataValidation
SE004_SOURCE 1_Data_Validation
SOURCE 1002WriteRejectFiles
Rejected (New Records):

Financial Totals,
Policies,
Policy Versions
TBD: Rejected (New Reco
Records,
Financial Totals,
Policies,
Policy Versions
SOURCE 1001EnrichmentSOURCE
1IncomingTable
Read (Rejects):
Records,
Financial Totals,
Policies,
Policy Versions
___________________
Rejected (New & Existing
Records,
Financial Totals,
Policies,
Policy Versions
SOURCE 1001LoadSAPOutTable
Written (New + Rejects):

Number Records,
Financial Totals
SOURCE 1002WriteOutgoingtables

Records,
Financial Totals,
Policies,
Policy Versions
SOURCE 2SOURCE 1004WriteSIFFIle

Records,
Financial Totals,
Policies,
Policy Versions
AIE002WriteSAPFile1

Records,
Financial Totals,
Policies,
Policy Versions
SE005_SOURCE 1_Enrichment
SE006_SIFF_File_Output
SE007_SAP_File_Output
Page 54 of 62
Solvency II
Job sequence name
SE009_SIFF_File_Validation
SE010_Load_SIFF_Receipt_Tables
SE011_SIFF_Data_Validation
SE011_SIFF_Data_Validation
SE012_LOB_Data_Mapping
213560877.doc
Job name
Metrics recorded
UKR002ValidatesFile
Read (New Records):

Records,
Financial Totals,
Policies,
Policy Versions
UKR001LoadIncomingTables
Loaded: Records,
Financial Totals,
Policies,
Policy Versions
SOURCE 2001DataValidation
Rejected (New Records):

Records,
Financial Totals,
Policies,
Policy Versions
SOURCE 2002WriteRejectFiles
Read (Rejects):
Records,
Financial Totals,
Policies,
Policy Versions
UKR002IncomingToDs
Read (Rejects):
Records,
Financial Totals,
Policies,
Policy Versions
UKR007UpdateIncoming
Rejected (New & Existing

Records,
Financial Totals,
Policies,
Policy Versions
UKR008LoadOutgoingTables

Records,
Financial Totals,
Policies,
Policy Versions
Read:
Records,
Financial Totals,
Policies,
Policy Versions
_________________
SE017_Load_Transaction_Tables_A
TDS007LoadCoinsuranceDB
Page 55 of 62
Loaded:
Records,
Financial Totals,
Policies,
Policy Versions
Solvency II
Job sequence name
213560877.doc
Job name
Metrics recorded
Read:
Records,
Financial Totals,
Policies,
Policy Versions
_________________
TDS008LoadReinsuranceContractDB
Loaded:
Records,
Financial Totals,
Policies,
Policy Versions
Read:
Records,
Financial Totals,
Policies,
Policy Versions
_________________
TDS009PolicyTransactionLoadDB
Loaded:
Records,
Financial Totals,
Policies,
Policy Versions
Read:
Records,
Financial Totals,
Policies,
Policy Versions
_________________
SE021_SOURCE
1_SAP_E2E_Reconciliation
SE021_SOURCE
SE021_SOURCE
TDS010LoadPolicyIncomeChargesDB
Loaded:
Records,
Financial Totals,
Policies,
Policy Versions
Read:
Records,
Financial Totals
Read:
Records,
Financial Totals
Read:
Records,
Financial Totals
Page 56 of 62
Solvency II
213560877.doc
7.3 Error Management

A requirement of the system is to log all errors that occur so that they can be analysed and acted on. The
information to be logged includes,
The Execution ID of the DataStage job that rSource 1ed the error
The Error Code
Process or Module reference to identify where in the job the error occurred
Error details. The error code defines the type of error for example "Invalid File Format" and this
field provides details for why the error was generated:
When recording errors the description must provide full details of the error as in the example above. This
data is recorded in the Error Log table.
The tables that support Error Logging are detailed below:
CTRL_ERROR (Static)
Column name
Data Type
Key
Description
ERROR_CODE
INT
PK
A unique code for the

error
ERROR_SHORT_CODE
ERROR_DESCRIPTION
VARCHAR2(500)
A short name for the

error e.g. "File
Validation Error"
VARCHAR2(1000)
A full description for

the error including
suggested remedial
action
Page 57 of 62
Solvency II
213560877.doc
CTRL_ERROR_LOG
Column name
Error_id
Error_Timestamp
Error_code
Job_execution_id
Data Type
Int
Key
Description
PK
An ID to uniquely
identify the error
event
DateTime
The date and time

error occurred as per
system date on the
ETL server
INT
FK from CTRL_ERROR
The unique ID of the

error that occurred.
FK to CTRL_Error
above
FK from
CTRL_BATCH_EXEC
The ID for the

executing batch, FK
to the
CTRL_Batch_Exec
table
INT
File_Name
Varchar(100)
(Optional) The name

of the source file that
is being processed.
Error_Detail
Varchar(500)
Details of the reason

for the error
Reject_File_Name
Varchar(100)
For data validation

and mapping errors
this is the name of
the reject file that
holds the rejected
data.
Job_ID
INT
FK from CTRL_Job_Defn
Page 58 of 62
Job ID on which error

occurred.
Solvency II
213560877.doc
Error Codes
Error Level
Error Code
Error Name
Error Description
Low
001
DateFormatError
A date in character format can not be converted to

a Date data type
Low
002
NumberFormatError
A number in character format can not be

converted to a Numeric
Low
003
DatetimeFormatError
A date/time in character format can not be

converted to a Datetime
Low
004
NotNullError
A mandatory field is null
Low
005
FinancialValidationError
The financial check on a source file, that header

value equals detail total, failed
Error Alert Process

An email alert will go to the user group (Financial.SystemsInterfaces@uk.Insurance Clientgroup.com)
providing the following information:
Error Code
Error Short Name
Job Execution ID
Job Name
Error Timestamp
Filename
Error Details
Only one email is to be sent per executing job.
7.4 Maintaining reference data

The reference data represents the TDS dimensions data and the TDS lookup data used in the lookups to
determine the Class Structure. This data will be initially loaded from csv files into the corresponding
database tables using the database tools.
The reference data will be maintained inside the database by updating the tables with new values. This
process needs to follow a change management policy.
7.5 Parameter Management

System Wide Parameters and Constants are defined in the ETL Configuration File. The table below
contains details of required parameters, however it is expected that more will be defined during build
process. Any new parameters defined are to be documented in the ETL Operators Manual.
Page 59 of 62
Solvency II
213560877.doc
Parameter Name
Description
proc.alert_email_address
The email address of where alert emails are to be

sent.
tds.high_date
The large future date used by Type II historical data.
folder.root
Root folder of the ETL process file system
folder.sil.reception
Reception folder for SIL data processing
folder.sil.archive
Archive folder for completed source files for SIL data

processing
folder.sil.reject
Reject folder for rejected source files for SIL data

processing
folder.sil.out
Output folder for generated SIL files
folder.sil.static
Folder for static files for SIL processing
folder.sil.scripts
Folder for any shell scripts used by SIL
folder.sil.temp
Temp folder for any working files required by SIL
folder.tds.reception
Reception folder for TDS data processing
folder.tds.archive
Archive folder for completed source files for TDS

data processing
folder.tds.reject
Reject folder for rejected source files for TDS data

processing
folder.tds.static
Folder for static files for TDS processing
folder.tds.scripts
Folder for any shell scripts used by TDS
folder.tds.temp
Temp folder for any working files required by TDS
db.sil.intermediate
All incoming and outgoing tables used during the

SIL intermediate process
db.sil.tds
All TDS tables
db.sil.databasename
Database name used for SIL.
db.tds.databasename
Database name used for TDS
Page 60 of 62
Solvency II
213560877.doc
7.6 File System Structure

This is the structure of the file system used for managing the data files in SIL and TDS.
Note: ADS will process data from database table to database table and so does not require use of flat files.
Area
Folder Name
Fully Qualified Path
Project
Root Folder
/<env>/data/SII_R01/
Inbound data
SOURCE 1 SIL Reception
/<env>/data/SII_R01/inbound/SOURCE
1
SOURCE 2 SIL Reception
/<env>/data/SII_R01/inbound/SOURCE
2
SAP SIL Reception
/<env>/data/SII_R01/inbound/SAP
SIL Archive
/<env>/data/SII_R01/inbound/archive
SIL Rejected Files
/<env>/data/SII_R01/inbound/reject
SIL Rejected Source
/<env>/data/SII_R01/inbound/reject_src
TDS Reception
/<env>/data/SII_R01/inbound/TDS
TDS Archive
/<env>/data/SII_R01/inbound/archive
TDS Rejected Files
/<env>/data/SII_R01/inbound/reject
Outbound data
SIL Data Output Folder
/<env>/data/SII_R01/outbound/
Static data
SIL Static Data
/<env>/data/SII_R01/static
TDS Static Data
/<env>/data/SII_R01/static
SIL Scripts
/<env>/data/SII_R01/scripts
TDS Scripts
/<env>/data/SII_R01/scripts
SIL Temp Space
/<env>/data/SII_R01/temp
TDS Temp Space
/<env>/data/SII_R01/temp
Scripts
Working area
Page 61 of 62
Solvency II
213560877.doc
7.7 FTP File locations for Dev and Test

Sever
Location
SAP Development Sever
Lnxddb01 10.109.191.33
SAP QA Test Server
Lnxqdb01 10.109.191.18
SAP QA Test Server (failover serve)
Lnxqdb02 10.109.191.20
SAP Production Server
Lnxpdb01 10.109.191.25
SAP Production Server (failover server)
Lnxpdb02 - 10.109.191.27
Page 62 of 62

Insurance Client S2 DE082 ETL Technical Specification v14

Hochgeladen von

Dokumentinformationen

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Insurance Client S2 DE082 ETL Technical Specification v14

Hochgeladen von

Copyright:

Verfügbare Formate

Solvency II

Insurance Client Logo

INSURANCE CLIENT Solvency II

Last Edit Date

CMI ETL Lead

Application Delivery Manager

IBM Solution Architect

Application Delivery Manager

Updated with the SAP E2E

Updated based on review

Updated with the review

Added the reconciliations for

Updated with the review

The sections 3.4 and 3.5 were

Updated the E2E Reconciliation

Updated with the review

Section 7.2 was updated to list

Updated for defect 319

Updated as per defect 424;

Load Cover Snapshot was

SAP Reference data metrics

Email from Mark Carter

SCS tables v02.xls

Email from Leon van Heerden

FTP Order for Unitech Files

Email from Mark Carter

INSURANCE CLIENT S2 DE05

6.2.1 Data Input..................................................................................................................................... 46

1.2 Scope for Increment 2

The key functional areas are:

Internal Reconciliation and Integrity Checking

SOURCE 1 SAP E2E Reconciliation

1.3 SIL Tables

1.4 External Dependencies

DataStage Server Availability

Source File Availability

2 Increment 2 ETL Processing Component

2.2 TDS to TDS Overview Diagram

SIFF File Receipt

2.3 Job Summary Table

Job Sequence used to get the SCS and load the

Job Sequence used to calculate the EP and UPR

Job Sequence used to get the SCS and load the

Job Sequence used to calculate the EP and UPR

Job Sequence used for the SOURCE 1 - SAP E2E

Job Sequence used for the SOURCE 1 TDS

Job Sequence used for the SOURCE 1 SAP

Job which writes the SOURCE 1 Incoming data into a

3 Internal Reconciliation Build

3.2 Structure and Mappings

Used to track progress of incoming SOURCE 1

Used to track progress of incoming SOURCE 1 detail

Used to assemble a TDS_POLICY_TRANSACTION

Used to assemble a SAP Detail record prior to

3.2.3 Data Output

Reconciliation Success file

Will contain the results of the reconciliation in case of

Reconciliation Error file

Will contain the results of the reconciliation in case of

3.3 SE022 SOURCE 1-TDS Reconciliation

3.3.2 Job Flow

3.3.3 Build Specification

DataStage component type/name