Sie sind auf Seite 1von 9

Informatica V9 Sizing Guide

Overview of Document
This document shows average sizing for V9 Installs at 3 different levels. The first is the size of
installed elements on the file system. The second is the runtime footprint of general V9 services for
all users. The last is the additional overhead in memory and disk of an individual users running
mappings.
The mappings contribution to disk/memory usage is usually the most critical and the most
difficult to average without particular details. The details below can be used as a basis of scaling
calculations based on number of concurrent mappings submitted to the server, transform usage in
the mapping and the data file input size in number of rows and columns.

Base on Disk Install Size


A typical Server Platform Install will take about 3.2 GB of Disk. This does not include disk
usage for reference data items listed in appendix 1 and 2. In addition it does not take account of the
database usage of a typical set of reference data items. The numbers below are for a full set of all
these content elements on the file system. Depending on individual customers usage appendix 1 and
2 may be used to estimate more exact disk sizing.
Server Platform Install Size

3.2 GB

Identity Reference Data

600 MB

Address Reference Data

4 GB

Reference Table Data

3 GB

This gives a rounded base figure of about 12 GB. This does not include additional customer reference
data or increases in Address Reference data which is amended as country postal authorities add
additional data.

General Runtime Memory Sizing


The below sizing is for an average running server with no disk/memory intensive mappings and no
loaded content
No

Service Name

Virtual Set

Working Set

1.

Admin Console

773K

133K

2.

MRS

1288K

407K

3.

Mapping Service

978K

254K

4.

Analyst Tool

702K

79K

This table shows the average sizes of the 4 V9 services of a typical configuration. The Virtual Set is
the total memory in virtual memory and the Working Set is the physically resident memory usage.

Address Validation Reference data


This data is loaded globally for all users. The customers configuration dictates which AV
reference files are loaded. The Address Validation file size guide may be used here to estimate
memory usage. The average size in memory of each loaded element is approximately the same as
the disk footprint.
For example if a user runs a mapping that uses the following reference file
United States

Batch/Interactive

533 MB

It would be expected that the process memory size will grow by 533MB approximately. It should be
noted that this memory cost is for the life time of the server and is a once off cost for the server and
all mappings run in the servers lifetime. The loaded Address Validation data is not unloaded even
when there are no current users for performance reasons.

User Created Mapping Memory and Disk footprint


This section is split into 3 types of Data Quality component. The Standard elements dont incur any
additional costs in memory or disk usage beyond its standard running size. The Dynamic components
are of 2 different types. Reference data based transforms which hold in-memory, the same
reference table lookup structures and Dynamic transforms that can include items like third party
engines, sort space or b-tree storage. The Dynamic transforms use both memory and disk that can
considerably depending on the data being processed.

Standard DQ Transformations
Comparison Transformation
Decision Transformation
Merge Transformation
None of the transforms have dynamic memory or disk usage that varies with the size of the
data being processed. All these components are referred to as passive since they process data rows
in small batches and send to the next component in the mapping immediately.

Reference Data Based Transformations


Case Convertor Transformation
Labeller Transformation

Parser Transformation
Standardiser Transformation
These transforms are all based around usage of reference data. While they are all passive in
that they process data immediately they have initialisation costs that increase memory based on
configuration. This memory usage makes them dynamic based on the transforms configuration but
not dynamic based on the number of rows presented for processing
While the reference data is managed in a database for editing, at runtime its held in
memory for performance. To optimise the throughput this in-memory storage is designed for speed
rather than space efficiency. The current list of reference tables available is around 3.5K so a list of
tables and in-memory sizes is not included. Each transform will have its own copy of the in-memory
reference data. To enable sizing the customer should take the number of bytes in each column of
the reference table and multiply it by the number of lines. This final calculation multiplied by 1.3 will
give an approximate guide to the in memory footprint.
For example a reference table with 10K rows and 6 columns with an average byte count per column
of 25 will give 10000 * 6 * 25 * 1.3 approximately 2M runtime memory usage. This runtime
memory cost is for the lifetime of the mapping. All in memory reference tables are freed when the
mapping is finished.

Dynamic DQ Transformations
All the following components have dynamic memory and disk usage. These components are referred
to as active and in general store large numbers of rows internally for block processing and have
memory/disk requirements that increase in-line with the volume of input rows and number of
corresponding columns per row
Address Validator Transformation
This component is treated in the General Runtime memory sizing section as it affects all
users as soon as the first mapping is run.
Association Transformation
This component makes extensive use of B-tree file based storage. Each column used in the
association will have its own b-tree and a general b-tree is used to store all the input data rows. The
Informatica b-tree is space efficient but not compressed. So the general sizing guideline here is as
follows,
Each association column is the total volume of data for each column * 20 bytes per input row
The general storage cache is the size of the input data set * 10 bytes per row will be the on disk
runtime cost.
An internal memory map of association ids and rows will be no larger than 20 bytes * the number of
rows
Sorter Based Transforms
Consolidation Transformation

Key Generator Transformation


These transforms all contain standard Informatica sort transforms. Currently they are all set
to auto. This is an internal configuration which attempts to give the transform as much memory as
possible without affecting system performance. When user wants more explicit control the sort
transform can be set with a memory limit on the maximum amount of main memory it can use to
sort data. The on disk temp size will grow as all data rows must be stored by the sort transform
Match Transformation
The match transform makes use of 2 different types of B-tree depending on its
configuration. When a user has configured a set of pass through ports and Identity matching both
types will be used. In general it can be assumed that the B-tree storage will not exceed in a
significant way the total size of disk the data would occupy if sitting outside the B-tree on the file
system.

Worked Sizing for US based Customer


Because any individual customer will have problem specific requirements the following
example shows how the data in this document may be applied to create more accurate sizing
estimates. The example shows the sizing for both disk and memory for a 4 user DQ installation using
US Address Validation, US Identity Matching and US Reference Dictionaries. While this number may
be small the variable elements of disk/memory usage only magnify when you have multiple users
concurrently using disk and memory intensive transforms. The transforms that have individual
requirements per mapping run are indicated in the document.

Base Server Disk Requirements

12 GB (Calculation shown above)

Base Memory Requirements

2 GB (Calculation shown above)

Assumption here is that a mapping without disk/memory sensitive components will add little
beyond the standard footprint. This will not be true with very complex mappings.
User 1 Running a matching mapping
Dual Source Identity with Source1 containing 1M rows and source2 containing 100K rows, 6
columns with 25 bytes per column, 20 columns of pass-through data with 25 bytes per column
This mapping will have 2 sorters from the key generation phase, 1 B-tree from matching, 1 B-tree
from Identity and internal memory usage for Identity and clustering
Disk Usage
B-tree 1 Identity = 1100000 * 6 * 25 = 165MB
B-tree 2 Pass-through = 1100000 * 20 * 25 = 550MB
Memory Usage = Internal storage for large number of transforms used for matching 10MB

User 2 Running an AV mapping


Single Source with Source1 containing 1M rows
This mapping will have minimal transforms but will load the all US AV validation reference data
United States

Batch/Interactive

533 MB

United States

GeoCoding

422 MB

United States

FastCompletion

380 MB

Total Disk added = 0


Memory Usage = 533 + 422 + 380 = 1.3 GB
User 3 Running Standardisation
Single Source with Source1 containing 10M rows
This mapping will have minimal transforms but will load 10 dictionaries to standardise
Assume each dictionary has 10K rows with 5 columns and 25 bytes average per column
Total Disk added = 0
Memory Usage 10000 * 5 * 25 * 1.3 = 1.6 MB per dictionary
Total Memory = 16MB
User 4 Running Association
Single Source with Source1 containing 10M rows and association running across 8 groups
This mapping will not have other matching transforms and will source data directly from a single
table. Each association key column will have a 10 byte key and there will be 10 additional columns
of row data each 50 bytes wide
Each Key column Btree will take 10M * ( 10 + 20) 300MB
General Storage will take 10M * ((8 * 10) + (10 * 50)) 5.8GB
Total Disk 300MB * 8 columns + 5.8GB = 8.2GB
Total Memory = 10M * 20 = 200MB
Total Additional Memory/Disk used by the 4 concurrently running mappings
Disk = 165MB + 550MB + 8200MB
= 8915MB
Memory = 10 MB + 1300MB + 16MB + 200MB = 1526MB

Summary
The data in this document estimates the standard disk and memory footprint of the V9
server. In addition the 2 tables shown at the end of the document will allow a user to minimise the
on disk footprint of the install if this is required. The Example sizing at the bottom of the document
shows how to estimate a mappings contribution to disk/memory by analysing the composition of the
mapping and each transforms contribution to disk/memory usage. The example also shows the
importance of factoring in the number of concurrent users and likely usage in defined the total peak
requirements of an individual installation.
Appendix 1
Address Validation Reference Data with On Disk size
Largest 50 files
United States

Batch/Interactive

533 MB

United Kingdom

FastCompletion

501 MB

United States

GeoCoding

422 MB

United States

FastCompletion

380 MB

United Kingdom

Batch/Interactive

306 MB

France

FastCompletion

210 MB

France

Batch/Interactive

153 MB

Argentina

FastCompletion

120 MB

Brazil

FastCompletion

104 MB

Germany

FastCompletion

102 MB

Germany

Batch/Interactive

99 MB

United Kingdom

Supplementary

94.5 MB

Italy

FastCompletion

92.9 MB

Argentina

Batch/Interactive

90 MB

Canada

FastCompletion

83.1 MB

India

FastCompletion

83.1 MB

India

Batch/Interactive

80 MB

Germany

GeoCoding

73.5 MB

Brazil

Batch/Interactive

73.3 MB

Italy

Batch/Interactive

66 MB

Canada

Batch/Interactive

61.8 MB

United Kingdom

GeoCoding

51.8 MB

Sweden

FastCompletion

49 MB

Mexico

FastCompletion

48.5 MB

Australia

FastCompletion

44.6 MB

Russian Federation

FastCompletion

44.3 MB

Mexico

Batch/Interactive

42.8 MB

Australia

Batch/Interactive

40.9 MB

Russian Federation

Batch/Interactive

40.5 MB

France

GeoCoding

39.7 MB

Portugal

FastCompletion

38.8 MB

Italy

GeoCoding

36.6 MB

Netherlands

FastCompletion

35.5 MB

Canada

GeoCoding

32.7 MB

China

FastCompletion

28.4 MB

Netherlands

Batch/Interactive

27.8 MB

Sweden

Batch/Interactive

27.4 MB

Spain

GeoCoding

25.6 MB

Australia

GeoCoding

25.4 MB

Spain

FastCompletion

23.7 MB

Chile

FastCompletion

23.4 MB

Netherlands

GeoCoding

22.7 MB

Portugal

Batch/Interactive

22.5 MB

China

Batch/Interactive

21.4 MB

Finland

GeoCoding

18.8 MB

Switzerland

FastCompletion

18.2 MB

Sweden

GeoCoding

17.8 MB

Chile

Batch/Interactive

16.8 MB

Belgium

FastCompletion

16.1 MB

Spain

Batch/Interactive

15.4 MB

The full list can be found at: http://www.addressdoctor.com/en/support/countrydownloadv5.asp

Appendix 2
Identity Based Matching Reference Data with On Disk Size
IM_japan_i.zip
IM_japan.zip
IM_japan_r.zip
IM_gaelic.zip
IM_canada.zip
IM_international.zip
IM_chinese_s.zip
IM_south_africa.zip
IM_uk.zip
IM_ireland.zip
IM_new_zealand.zip
IM_australia.zip
IM_usa.zip
IM_arabic_m.zip
IM_indonesia.zip
IM_cyrillic.zip
IM_arabic_r.zip
IM_singapore.zip
IM_india.zip
IM_chinese_t.zip
IM_aml.zip
IM_greek_l.zip
IM_switzerland.zip
IM_france.zip
IM_philippines.zip
IM_luxembourg.zip
IM_belgium.zip
IM_germany.zip
IM_brasil.zip
IM_portugal.zip
IM_korean_r.zip
IM_italy.zip
IM_turkey.zip
IM_hk_r.zip
IM_sweden.zip
IM_czech.zip

86,222,167
86,222,153
15,754,935
9,237,372
8,933,319
5,303,974
4,955,588
4,260,152
4,241,637
4,241,357
4,200,805
4,153,252
4,134,750
3,893,388
3,494,046
3,022,104
2,980,176
2,505,578
2,321,418
2,189,993
2,083,153
2,057,442
2,028,497
1,950,898
1,896,332
1,812,614
1,696,864
1,604,137
1,596,925
1,596,786
1,588,819
1,554,842
1,552,887
1,542,915
1,528,272
1,525,846

IM_netherlands.zip
IM_taiwan_r.zip
IM_denmark.zip
IM_slovakia.zip
IM_malaysia.zip
IM_thai_r.zip
IM_spain.zip
IM_chinese_r.zip
IM_colombia.zip
IM_argentina.zip
IM_indo_chin_r.zip
IM_chile.zip
IM_peru.zip
IM_vietnam_r.zip
IM_puerto_rico.zip
IM_mexico.zip
IM_thai.zip
IM_finland.zip
IM_norway.zip
IM_poland.zip
IM_greek.zip
IM_hungary.zip
IM_estonia.zip
IM_korean.zip
IM_ofac.zip
IM_hebrew.zip
IM_chinese_i.zip
IM_arabic.zip

1,476,954
1,473,532
1,473,231
1,458,393
1,447,577
1,443,929
1,438,526
1,431,129
1,414,047
1,413,962
1,410,620
1,400,965
1,389,800
1,379,744
1,372,143
1,344,656
1,279,607
1,273,884
1,273,795
1,261,906
1,247,548
1,205,908
1,092,791
821,290
759,006
754,978
544,844
297,401

Das könnte Ihnen auch gefallen