Data Warehouse Concepts: Avinash Kanumuru Diya Jana Debyajit Majumder

Data Warehouse Concepts
Avinash Kanumuru Diya Jana Debyajit Majumder
2009 Wipro Ltd - Confidential 2009 Wipro Ltd - Confidential
Content
1 2 3 4 5 An Overview of Data Warehouse Data Warehouse Architecture Data Modeling for Data Warehouse Overview of Data Cleansing Data Extraction , Transformation , Load
Content
Content [contd]
6 7 8 Metadata Management OLAP Data Warehouse Testing
An Overview
Understanding What is a Data Warehouse
10
What is Data Warehouse?

Definitions of Data Warehouse A data warehouse is a subject-oriented, integrated, nonvolatile, time-variant collection of data in support of management's decisions. WH Inmon Data Warehouse is a repository of data summarized or aggregated in simplified form from operational systems. End user orientated data access and reporting tools let user get at the data for decision support Babcock A data warehouse is a relational database a copy of transaction data specifically structured for query and analysis Ralph Kimball In simple: Data warehousing is collection of data from different systems, which helps in Business Decisions, Analysis and Reporting.
11
Data Warehouse def. by WH Inmon

A common way of introducing data warehousing is to refer to the characteristics of a data warehouse as set forth by William Inmon: Subject Oriented Data that gives information about a particular subject instead of about a company's ongoing operations. Integrated Data that is gathered into the data warehouse from a variety of sources and merged into a coherent whole. Nonvolatile Data is stable in a data warehouse. More data is added but data is never removed. This enables management to gain a consistent picture of the business. Time Variant In order to discover trends in business, analysts need large amounts of data. This is very much in contrast to online transaction processing ( OLTP ) systems, where performance requirements demand that historical data be moved to an archive. All data in the data warehouse is identified with a particular time period.
12
Data Warehouse Architecture
What makes a Data Warehouse
13
Components of Warehouse
Source Tables : These are real-time, volatile data in relational databases for transaction processing (OLTP). These can be any relational databases or flat files. ETL Tools: To extract, cleansing, transform (aggregates, joins) and load the data from sources to target. Maintenance and Administration Tools: To authorize and monitor access to the data, set-up users. Scheduling jobs to run on offshore periods. Modeling Tools: Used for data warehouse design for high-performance using dimensional data modeling technique, mapping the source and target files. Databases: Target databases and data marts, which are part of data warehouse. These are structured for analysis and reporting purposes. End - user tools for analysis and reporting: get the reports and analyze the data from target tables. Different types of Querying, Data Mining, OLAP tools are used for this purpose.
14

This is a basic design, where there are source files, which are loaded to a warehouse and users query the data for different purposes.
This has a staging area, where the data after cleansing, transforming is loaded and tested here. Later is directly loaded to the target database/warehouse. Which is divided to data marts and can be accessed by different users for their reporting and analyzing purposes.
15 22
16 22
2009 Wipro Ltd - Confidential Wipro 2222 Ltd - Confidential

17 22
Data Modeling
Effective way of using a Data Warehouse
18
Data Modeling
Commonly E-R Data Model is used in OLTP, In OLAP Dimensional Data Model is used commonly. E-R (Entity-Relationship) Data Model
Entity : Object that can be observed and classified based on its properties and characteristics. Like employee, book, student Relationship : relating entities to other entities.

Different Perceptive of Data Modeling.

o Conceptual Data Model o Logical Data Model o Physical Data Model o
Types of Dimensional Data Models most commonly used:

o Star Schema o Snowflake Schema
19
Terms used in Dimensional Data Model

To understand dimensional data modeling, let's define some of the terms commonly used in this type of modeling: Dimension: A category of information. For example, the time dimension. Attribute: A unique level within a dimension. For example, Month is an attribute in the Time Dimension. Hierarchy: The specification of levels that represents relationship between different attributes within a dimension. For example, one possible hierarchy in the Time dimension is Year Quarter Month Day. Fact Table: A table that contains the measures of interest. Lookup Table: It provides the detailed information about the attributes. For example, the lookup table for the Quarter attribute would include a list of all of the quarters available in the data warehouse. Surrogate Keys: To avoid the data integrity, surrogate keys are used. They are helpful for Slow Changing Dimensions and act as index/primary keys.
20 22
A dimensional model includes fact tables and lookup tables. Fact tables connect to one or more lookup tables, but fact tables do not have direct relationships to one another. Dimensions and hierarchies are represented by 2009 Wipro Ltd Confidential lookup tables. Attributes --are the non-key columns in the lookup tables. 2222 Wipro Ltd Confidential

21 22
Data Modeling



22 2 2

23 22
A dimensional model includes fact tables and lookup tables. Fact tables connect to one or more lookup tables, but fact tables do not have direct relationships to one another. Dimensions and hierarchies are represented by 2009 Wipro Ltd Confidential Wipro lookup tables. Attributes --are the non-key columns in the lookup tables. 2222 Ltd Confidential
Star Schema
Dimension Table
product prodId p1 p2 name price bolt 10 nut 5
Dimension Table
store storeId city c1 nyc c2 sfo c3 la
Fact Table
sale oderId date o100 1/7/97 o102 2/7/97 105 3/8/97 custId 53 53 111 prodId p1 p2 p1 storeId c1 c1 c3 qty 1 2 5 amt 12 11 50
Dimension Table
customer custId 53 81 111 name joe fred sally address 10 main 12 main 80 willow city sfo sfo la
24 22
Snowflake Schema
Dimension Table Fact Table
store storeId s5 s7 s9 cityId sfo sfo la tId t1 t2 t1 mgr joe fred nancy sType tId t1 t2 cityId sfo la size small large pop 1M 5M location downtown suburbs regId north south
Dimension Table
city
The star and snowflake schema are most commonly found in dimensional data warehouses and data marts where speed of data retrieval is more important than the efficiency of data manipulations. As such, the tables in these schema are not normalized much, and are frequently designed at a level of normalization short of third normal form.
region
regId north south
name cold region warm region
25
Overview of Data Cleansing
26
The Need For Data Quality

Difficulty in decision making Time delays in operation Organizational mistrust Data ownership conflicts Customer attrition Costs associated with error detection error rework customer service fixing customer problems
27
Understand Information Flow In Organization
Identify authoritative data sources Identify authoritative data sources Interview Employees & Customers Interview Employees & Customers Data Entry Points Data Entry Points Cost of bad data Cost of bad data
Id e n tify P o te n tia l Id e n tify P o te n tia l P ro b le m A re a s & A s s e s P ro b le m A re a s & A ss e s Im p a ct Im p a ct
M e a s u re Q u a lity O f M e a s u re Q u a lity O f D a ta D a ta C le a n & L o a d C le a n & L o a d D a ta D a ta
Use business rule discovery tools to identify data with Use business rule discovery tools to identify data with
inconsistent, missing, incomplete, duplicate or incorrect inconsistent, missing, incomplete, duplicate or incorrect values values
Use data cleansing tools to clean data at the source Use data cleansing tools to clean data at the source Load only clean data into the data warehouse Load only clean data into the data warehouse
C o n tin u o u s M o n ito rin g C o n tin u o u s M o n ito rin g
Schedule Periodic Cleansing of Source Data Schedule Periodic Cleansing of Source Data
Id e n tify A re a s o f Id e n tify A re a s o f Im p ro v e m e n t Im p ro v e m e n t
Identify & Correct Cause of Defects Identify & Correct Cause of Defects Refine data capture mechanisms at source Refine data capture mechanisms at source Educate users on importance of DQ Educate users on importance of DQ
28
Customized Programs Strengths: Addresses specific needs No bulky one time investment Limitations Tons of Custom programs in different environments are difficult to manage Minor alterations demand coding efforts Data Quality Assessment tools Strength Provide automated assessment Limitation No measure of data accuracy
29
Business Rule Discovery tools Strengths Detect Correlation in data values Can detect Patterns of behavior that indicate fraud Limitations Not all variables can be discovered Some discovered rules might not be pertinent There may be performance problems with large files or with many fields.

Data Reengineering & Cleansing tools Strengths Usually are integrated packages with cleansing features as Add-on Limitations Error prevention at source is usually absent The ETL tools have limited cleansing facilities
30
Business Rule Discovery Tools Integrity Data Reengineering Tool from Vality Technology Trillium Software System from Harte -Hanks Data Technologies Migration Architect from DB Star Data Reengineering & Cleansing Tools Carlton Pureview from Oracle ETI-Extract from Evolutionary Technologies PowerMart from Informatica Corp Sagent Data Mart from Sagent Technology Data Quality Assessment Tools Migration Architect, Evoke Axio from Evoke Software Wizrule from Wizsoft Name & Address Cleansing Tools Centrus Suite from Sagent I.d.centric from First Logic

31
Data Extraction, Transformation, Load
32
ETL Architecture
33
ETL Architecture

Data Extraction:
Rummages through a file or database Uses some criteria for selection Identifies qualified data and Transports the data over onto another file or database
D a ta tra n sfo rm a ti n o
I te g ra ti g d i m i a r d a ta typ e s n n ssi l C h a n g i g co d e s n A d d i g a ti e a ttri u te n m b S u m m a ri n g d a ta zi C a l l ti g d e ri d va l e s cu a n ve u R e n o rm a l zi g d a ta i n
Data Extraction Cleanup

Restructuring of records or fields Removal of Operational-only data Supply of missing field values Data Integrity checks Data Consistency and Range checks, etc...
Data loading
Initial and incremental loading Updation of metadata
34
Companies have valuable data lying around throughout their networks that needs to be moved from one place to another. The data lies in all sorts of heterogeneous systems,and therefore in all sorts of formats. To solve the problem, companies use extract, transform and load (ETL) software.
The data used in ETL processes can come from any source: a mainframe application, an ERP application, a CRM tool, a flat file, and an Excel spreadsheet.

35
Major components involved in ETL Processing
36
Design manager Lets developers define source-to-target mappings, transformations, process flows, and jobs Meta data management Provides a repository to define, document, and manage information about the ETL design and runtime processes Extract The process of reading data from a database. Transform The process of converting the extracted data Load The process of writing the data into the target database. Transport services ETL tools use network and file protocols to move data between source and target systems and in-memory protocols to move data between ETL run-time components. Administration and operation ETL utilities let administrators schedule, run, monitor ETL jobs, log all events, manage errors, recover from failures, reconcile outputs with source systems
37
ETL Tools
Provides facility to specify a large number of transformation rules with a GUI Generate programs to transform data Handle multiple data sources Handle data redundancy Generate metadata as output Most tools exploit parallelism by running on multiple low-cost servers in multi-threaded environment
ETL Tools - Second-Generation PowerCentre/Mart from Informatica Data Mart Solution from Sagent Technology DataStage from Ascential
38
Metadata Management
39
What Is Metadata?

Metadata is Information...
That describes the WHAT, WHEN, WHO, WHERE, HOW of the data warehouse About the data being captured and loaded into the Warehouse Documented in IT tools that improves both business and technical understanding of data and data-related processes
40
Importance Of Metadata
Locating Information Time spent in looking for information . How often information is found? What poor decisions were made based on the incomplete information? How much money was lost or earned as a result? Interpreting information How many times have businesses needed to rework or recall products? What impact does it have on the bottom line ? How many mistakes were due to misinterpretation of existing How much interpretation results form too much metadata? How much time is spent trying to determine if any of the metadata is accurate? Integrating information How various data perspectives connect together? How much time is spent trying to figure out that? How much does the inefficiency and lack of metadata affect decision making documentation?
41
Requirements for DW Metadata Management

Provide a simple catalogue of business metadata descriptions and views Document/manage metadata descriptions from an integrated development environment Enable DW users to identify and invoke pre-built queries against the data stores Design and enhance new data models and schemas for the data warehouse Capture data transformation rules between the operational and data warehousing databases Provide change impact analysis, and update across these technologies
42
Consumers of Metadata
Technical Users Warehouse administrator Application developer Business Users -Business metadata Meanings Definitions Business Rules Software Tools Used in DW life-cycle development Metadata requirements for each tool must be identified The tool-specific metadata should be analysed for inclusion in the enterprise metadata repository Previously captured metadata should be electronically transferred from the enterprise metadata repository to each individual tool
43
44
Trends in the Metadata Management Tools

Third Party Bridging Tools Oracle Exchange
Technology of choice for a long list of repository, enterprise and workgroup vendors
Reischmann-Informatik-Toolbus
Features include facilitation of selective bridging of metadata
Ardent Software/ Dovetail Software -Interplay

Hub and Spoke solution for enabling metadata interoperability Ardent focussing on own engagements, not selling it as independent product
Informix's Metadata Plug-ins

Available with Ardent Datastage version 3.6.2 free of cost for Erwin, Oracle Designer, Sybase Powerdesigner, Brio, Microstrategy
45

Metadata Repositories IBM, Oracle and Microsoft to offer free or near-free basic repository services Enable organisations to reuse metadata across technologies Integrate DB design, data transformation and BI tools from different vendors Multi-tool vendors taking a bridged or federated rather than integrated approach to sharing metadata Both IBM and Oracle have multiple repositories for different lines of products e.g., One for AD and one for DW, with bridges between them
46

Metadata Interchange Standards CDIF (CASE Data Interchange Format) OMG (Object Management Group)-CWM
Most frequently used interchange standard Addresses only a limited subset of metadata artifacts XML-addresses context and data meaning, not presentation Can enable exchange over the web employing industry standards for storing and sharing programming data Will allow sharing of UML and MOF objects b/w various development tools and repositories Based on XML/UML standards Promoted by Microsoft Along With 20 partners including Object Management Group (OMG), Oracle Carleton Group, CAPLATINUM Technology (Founding Member), Viasoft
MDC (Metadata Coalition)
47



48

49

50
OLAP
51
Agenda
OLAP Definition Distinction between OLTP and OLAP MDDB Concepts Implementation Techniques Architectures Features Representative Tools
09/02/2012
52
52
OLAP: On-Line Analytical Processing

OLAP can be defined as a technology which allows the users to view the aggregate data across measurements (like Maturity Amount, Interest Rate etc.) along with a set of related parameters called dimensions (like Product, Organization, Customer, etc.) Used interchangeably with BI Multidimensional view of data is the foundation of OLAP Users :Analysts, Decision makers
09/02/2012
53
53
Distinction between OLTP and OLAP

OLTP System Source of data OLAP System
Operational data; OLTPs Consolidation data; OLAP are the original source data comes from the of the data various OLTP databases To control and run fundamental business tasks A snapshot of ongoing business processes Decision support Multi-dimensional views of various kinds of business activities
Purpose of data What the data reveals Inserts and Updates
Short and fast inserts Periodic long-running and updates initiated by batch jobs refresh the end users data
54
09/02/2012
54
MDDB Concepts
A multidimensional database is a computer software system designed to allow for efficient and convenient storage and retrieval of data that is intimately related and stored, viewed and analyzed from different perspectives (Dimensions).

A hypercube represents a collection of multidimensional data. The edges of the cube are called dimensions Individual items within each dimensions are called members
55
RDBMS v/s MDDB:

VOL . MINI VAN BLUE MINI VAN BLUE MINI VAN BLUE MINI VAN RED MINI VAN RED MINI VAN RED MINI VAN WHITE MINI VAN WHITE MINI VAN WHITE SPORTS COUPE SPORTS COUPE SPORTS COUPE SPORTS COUPE SPORTS COUPE SPORTS COUPE SPORTS COUPE SPORTS COUPE SPORTS COUPE SEDAN BLUE SEDAN BLUE SEDAN BLUE ... MODEL Clyde 6 Gleason 3 Carr 2 Clyde 5 Gleason3 Carr 1 Clyde 3 Gleason1 Carr 4 BLUE Clyde 3 BLUE Gleason BLUE Carr 3 RED Clyde 4 RED Gleason3 RED Carr 6 WHITE Clyde 2 WHITE Gleason3 WHITE Carr 5 Clyde 4 Gleason 3 Carr 2 ...
Increased Complexity...
COLOR DEALER
Relational DBMS
MDDB
Sales Volumes
M O D E L
Mini Van Coupe Sedan Blue Red White Carr Gleason Clyde
DEALERSHIP
COLOR
27 x 4 = 108 cells
56
3 x 3 x 3 = 27 cells
Benefits of MDDB over RDBMS

Ease of Data Presentation & Navigation A great deal of information is gleaned immediately upon direct inspection of the array User is able to view data along presorted dimensions with data arranged in an inherently more organized, and accessible fashion than the one offered by the relational table. Storage Space Very low Space Consumption compared to Relational DB Performance Gives much better performance . Relational DB may give comparable results only through database tuning (indexing, keys etc), which may not be possible for ad-hoc queries. Ease of Maintenance No overhead as data is stored in the same way it is viewed. In Relational DB, indexes, sophisticated joins etc. are used which require considerable storage and maintenance
09/02/2012
57
57
Issues with MDDB
Sparsity
- Input data in applications are typically sparse

-Due to Sparsity -Due to Summarization
-Increases with increased dimensions Data Explosion Performance

-Doesnt perform better than RDBMS at high data volumes (>20-30 GB)
09/02/2012
58
58
Issues with MDDB - Sparsity Example
If dimension members of different dimensions do not interact , then blank cell is left behind.
Employee Age
21 19 63 31 27 56 45 41 19
31 41 23 01 14 54 03 12 33
LAST NAME EMP# AGE SMITH 01 21 REGAN 12 19 Sales Volumes FOX 31 63 WELD 14 Van 6 5 4 Miini 31 M O KELLY 54 Coupe 27 D E LINK 03 56 3 5 5 L KRANZ 41 Sedan 4 3 2 45 LUCUS 33 41 Blue Red White WEISS 23 19 COLOR
Smith Regan Fox
L A S T N A M E
Weld Kelly Link Kranz Lucas Weiss
EMPLOYEE #
09/02/2012
59
59
OLAP Features
Calculations applied across dimensions, through hierarchies and/or across members Trend analysis over sequential time periods, What-if scenarios. Slicing / Dicing subsets for on-screen viewing Rotation to new dimensional comparisons in the viewing area Drill-down/up along the hierarchy Reach-through / Drill-through to underlying detail data
09/02/2012
60
60
Features of OLAP - Rotation

Complex Queries & Sorts in Relational environment translated to simple rotation.
Sales Volumes
M O D E L
Mini Van Coupe
6 3 4
Blue
5 5 3
Red
4 5 2
White
C O L O R ( ROTATE 90 )
o
Blue Red White
6 5 4
3 5 5
MODEL
4 3 2
Sedan
Sedan
Mini Van Coupe
COLOR
View #1
View #2
09/02/2012
61
2 dimensional array has 2 views.

61

Sales Volumes
M O D E L
C O L O R
Blue
Red White Sedan Coupe Mini Van Carr Gleason Clyde
C O L O R
Blue
Red White Carr Gleason Clyde Mini Van Coupe Sedan
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
( ROTATE 90 )
DEALERSHIP
DEALERSHIP
MODEL
View #1
D E A L E R S H I P D E A L E R S H I P
View #2
View #3
Carr Gleason Clyde White Red Blue Mini Van Coupe Sedan
Carr Gleason Blue Red White
Clyde Mini Van Coupe Sedan
M O D E L
Mini Van Coupe Sedan Clyde Gleason Carr Blue Red White
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
MODEL
COLOR
COLOR
View #4
View #5
View #6
09/02/2012
62

62
Features of OLAP - Slicing / Filtering

MDDB allows end user to quickly slice in on exact view of the data required.
Sales Volumes
M O D E L
Mini Van
Mini Van Coupe Normal Metal Blue Blue
Coupe
Carr Clyde
Carr Clyde
Normal Blue
Metal Blue
DEALERSHIP
COLOR
09/02/2012
63
63
Features of OLAP - Drill Down / Up
ORGANIZATION DIMENSION
REGION Chicago Clyde Gleason Midwest St. Louis Carr Levi Gary Lucas Bolton
DISTRICT DEALERSHIP
Sales at region/District/Dealership Level
Moving Up and moving down in a hierarchy is referred to as drill-up / roll-up and drill-down
09/02/2012
64
64
OLAP Reporting - Drill Down
Inflows ( Region , Year)

200 150 Inflows 100 ($M) 50 0 Year Year 1999 2000 Years
East West Central
09/02/2012
65
65
Inflows ( Region , Year - Year 1999)

90 80 70 60 50 Inflows ( $M) 40 30 20 10 0
East West Central 1st Qtr 2nd Qtr 3rd Qtr 4th Qtr Year 1999
Drill-down from Year to Quarter

09/02/2012
66
66
Inflows ( Region , Year - Year 1999 - 1st Qtr)

20 15 Inflows ( $M 10 ) 5 0 January February March Year 1999 East West Central
Drill-down from Quarter to Month
67
Implementation Techniques -OLAP Architectures
MOLAP - Multidimensional OLAP

Multidimensional Databases for database and application logic layer
ROLAP - Relational OLAP

Access Data stored in relational Data Warehouse for OLAP Analysis. Database and Application logic provided as separate layers
HOLAP - Hybrid OLAP

OLAP Server routes queries first to MDDB, then to RDBMS and result processed on-the-fly in Server
DOLAP - Desk OLAP

Personal MDDB Server and application on the desktop
09/02/2012
68
68
MOLAP - MDDB storage
OLAP
Cube
OLAP Calculation Engine
Web Browser
OLAP Tools
OLAP Applications
09/02/2012
69
69
MOLAP - Features
Powerful analytical capabilities (e.g., financial, forecasting, statistical) Aggregation and calculation capabilities Read/write analytic applications Specialized data structures for
Maximum query performance. Optimum space utilization.
09/02/2012
70
70
ROLAP - Standard SQL storage
Relational DW
MDDB - Relational Mapping
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
71
71
ROLAP - Features
Three-tier hardware/software architecture:
GUI on client; multidimensional processing on mid-tier server; target database on database server Processing split between mid-tier & database servers
Ad hoc query capabilities to very large databases DW integration Data scalability
09/02/2012
72
72
HOLAP - Combination of RDBMS and MDDB

OLAP Cube
Any Client
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
73
73
HOLAP - Features
RDBMS used for detailed data stored in large databases MDDB used for fast, read/write OLAP analysis and calculations Scalability of RDBMS and MDDB performance Calculation engine provides full analysis features Source of data transparent to end user
09/02/2012
74
74
Architecture Comparison
MOLAP
Definition MDDB OLAP = Transaction level data + summary in MDDB Good Design 3 10 times High (May go beyond control. Estimation is very important) Fast - (Depends upon the size of the MDDB)
ROLAP
Relational OLAP = Transaction level data + summary in RDBMS No Sparsity To the necessary extent
HOLAP
Hybrid OLAP = ROLAP + summary in MDDB Sparsity exists only in MDDB part To the necessary extent
Data explosion due to Sparsity Data explosion due to Summarization Query Execution Speed
Slow
Optimum - If the data is fetched from RDBMS then its like ROLAP otherwise like MOLAP. High: RDBMS + disk space + MDDB Server cost Large transactional data + frequent summary analysis
Cost
Medium: MDDB Server + large disk space cost Small transactional data + complex model + frequent summary analysis
Low: Only RDBMS + disk space cost Very large transactional data & it needs to be viewed / sorted
Where to apply?
09/02/2012
75
75
Representative OLAP Tools:
Oracle Express Products Hyperion Essbase Cognos -PowerPlay Seagate - Holos SAS
Micro Strategy - DSS Agent Informix MetaCube Brio Query Business Objects / Web Intelligence
76
09/02/2012
76
Sample OLAP Applications
Sales Analysis Financial Analysis Profitability Analysis Performance Analysis Risk Management Profiling & Segmentation Scorecard Application NPA Management Strategic Planning Customer Relationship Management (CRM)
09/02/2012
77
77
Data Warehouse Testing
78
Data Warehouse Testing Overview

There is an exponentially increasing cost associated with finding software defects later in the development lifecycle. In data warehousing, this is compounded because of the additional business costs of using incorrect data to make critical business decisions The methodology required for testing a Data Warehouse is different from testing a typical transaction system
79
Difference In Testing Data warehouse and Transaction System

Data warehouse testing is different on the following counts: User-Triggered vs. System triggered Volume of Test Data Possible scenarios/ Test Cases Programming for testing challenge
80
Difference In Testing Data warehouse and Transaction System.

User-Triggered vs. System triggered In data Warehouse, most of the testing is system triggered. Most of the production/Source system testing is the processing of individual transactions, which are driven by some input from the users (Application Form, Servicing Request.). There are very few test cycles, which cover the system-triggered scenarios (Like billing, Valuation.)
81

Volume of Test Data The test data in a transaction system is a very small sample of the overall production data. Data Warehouse has typically large test data as one does try to fillup maximum possible combination of dimensions and facts. Possible scenarios/ Test Cases In case of Data Warehouse, the permutations and combinations one can possibly test is virtually unlimited due to the core objective of Data Warehouse is to allow all possible views of data.
82

Programming for testing challenge In case of transaction systems, users/business analysts typically test the output of the system. In case of data warehouse, most of the 'Data Warehouse data Quality testing' and ETL testing is done at backend by running separate stand-alone scripts. These scripts compare pre-Transformation to post Transformation of data.
83
Data Warehouse Testing Process

Data-Warehouse testing is basically divided into two parts : 'Back-end' testing where the source systems data is compared to the end-result data in Loaded area 'Front-end' testing where the user checks the data by comparing their MIS with the data displayed by the end-user tools like OLAP. Testing phases consists of : Requirements testing Unit testing Integration testing Performance testing Acceptance testing
84
Requirements testing
The main aim for doing Requirements testing is to check stated requirements for completeness. Requirements can be tested on following factors. Are the requirements Complete? Are the requirements Singular? Are the requirements Ambiguous? Are the requirements Developable? Are the requirements Testable?
85
Unit testing for data warehouses is WHITEBOX. It should check the ETL procedures/mappings/jobs and the reports developed. Unit testing the ETL procedures:
Whether ETLs are accessing and picking up right data from right source. All the data transformations are correct according to the business rules and data warehouse is correctly populated with the transformed data. Testing the rejected records that dont fulfil transformation rules.
86
Unit Testing
Unit Testing
Unit Testing the Report data:
Verify Report data with source: Data present in a data warehouse will be stored at an aggregate level compare to source systems. QA team should verify the granular data stored in data warehouse against the source data available Field level data verification: QA team must understand the linkages for the fields displayed in the report and should trace back and compare that with the source systems Derivation formulae/calculation rules should be verified
87
Integration Testing
Integration testing will involve following: Sequence of ETLs jobs in batch. Initial loading of records on data warehouse. Incremental loading of records at a later date to verify the newly inserted or updated data. Testing the rejected records that dont fulfil transformation rules. Error log generation
88
Performance Testing
Performance Testing should check for :

ETL processes completing within time window. Monitoring and measuring the data quality issues. Refresh times for standard/complex reports.
89
Acceptance testing
Here the system is tested with full functionality and is expected to function as in production. At the end of UAT, the system should be acceptable to the client for use in terms of ETL process integrity and business functionality and reporting.
90
Questions
91
Thank You
92
93
Content
94
Content [contd]
95
An Overview
96

97

98
99
100
Content
101
Content [contd]
102
An Overview
103

104

105
106
107
Content
108
Content [contd]
109
An Overview
110

111

112
113
114

115 222
Data Modeling
116
Data Modeling



117

118 222
Star Schema
Dimension Table
Dimension Table
Fact Table
Dimension Table
119 222
Snowflake Schema
Dimension Table
city
region
regId north south
120
121

122
123
124

125

126
127
ETL Architecture
128
ETL Architecture

Data Extraction:

Data loading
129

130
131
132
ETL Tools
133
Metadata Management
134
What Is Metadata?

135
136

137
138



139

140

141
OLAP
142
Agenda
09/02/2012
143
143

09/02/2012
144
144

145
09/02/2012
145
MDDB Concepts

146
RDBMS v/s MDDB:

COLOR DEALER
Relational DBMS
MDDB
Sales Volumes
M O D E L
DEALERSHIP
COLOR
27 x 4 = 108 cells
147
3 x 3 x 3 = 27 cells

09/02/2012
148
148
Issues with MDDB
Sparsity


09/02/2012
149
149
Employee Age
21 19 63 31 27 56 45 41 19
31 41 23 01 14 54 03 12 33
Smith Regan Fox
L A S T N A M E
EMPLOYEE #
09/02/2012
150
150
OLAP Features
09/02/2012
151
151

Sales Volumes
M O D E L
Mini Van Coupe
6 3 4
Blue
5 5 3
Red
4 5 2
White
o
Blue Red White
6 5 4
3 5 5
MODEL
4 3 2
Sedan
Sedan
Mini Van Coupe
COLOR
View #1
View #2
09/02/2012
152

152

Sales Volumes
M O D E L
C O L O R
Blue
C O L O R
Blue
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
( ROTATE 90 )
DEALERSHIP
DEALERSHIP
MODEL
View #1
View #2
View #3
M O D E L
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
MODEL
COLOR
COLOR
View #4
View #5
View #6
09/02/2012
153

153

Sales Volumes
M O D E L
Mini Van
Coupe
Carr Clyde
Carr Clyde
Normal Blue
Metal Blue
DEALERSHIP
COLOR
09/02/2012
154
154
DISTRICT DEALERSHIP
09/02/2012
155
155

East West Central
09/02/2012
156
156

90 80 70 60 50 Inflows ( $M) 40 30 20 10 0

09/02/2012
157
157

158


HOLAP - Hybrid OLAP

DOLAP - Desk OLAP

09/02/2012
159
159
OLAP
Cube
Web Browser
OLAP Tools
OLAP Applications
09/02/2012
160
160
MOLAP - Features
09/02/2012
161
161
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
162
162
ROLAP - Features
09/02/2012
163
163

OLAP Cube
Any Client
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
164
164
HOLAP - Features
09/02/2012
165
165
MOLAP
ROLAP
HOLAP
Slow
Cost
Where to apply?
09/02/2012
166
166
167
09/02/2012
167
09/02/2012
168
168
169

170

171

172

173

174

175
176
177
Unit Testing
Unit Testing
178
Integration Testing
179
Performance Testing

180
Acceptance testing
181
Questions
182
Thank You
183
184

185 222
Data Modeling
186
Data Modeling



187

188 222
Star Schema
Dimension Table
Dimension Table
Fact Table
Dimension Table
189 222
Snowflake Schema
Dimension Table
city
region
regId north south
190
191

192
193
194

195

196
197
ETL Architecture
198
ETL Architecture

Data Extraction:

Data loading
199

200
201
202
203
204
Content
205
206
207
Content
208
Content [contd]
209
An Overview
210

211

212
213
214

215 222
216 222

217 222
Data Modeling
218
Data Modeling



219

220 222

221 22 2
Data Modeling



222 22 2

223 222
Star Schema
Dimension Table
Dimension Table
Fact Table
Dimension Table
224 222
Snowflake Schema
Dimension Table
city
region
regId north south
225
226

227
228
229

230

231
232
ETL Architecture
233
ETL Architecture

Data Extraction:
Integrating dissimilar data types Changing codes Adding a time attribute Summarizing data Calculating derived values Renormalizing data

Data loading
234

235
236
237
ETL Tools
238
Metadata Management
239
What Is Metadata?

240
241

242
243
244



245

246

247



248

249

250
OLAP
251
Agenda
09/02/2012
252
252

09/02/2012
253
253

254
09/02/2012
254
MDDB Concepts

255
RDBMS v/s MDDB:

COLOR DEALER
Relational DBMS
MDDB
Sales Volumes
M O D E L
DEALERSHIP
COLOR
27 x 4 = 108 cells
256
3 x 3 x 3 = 27 cells

09/02/2012
257
257
Issues with MDDB
Sparsity


09/02/2012
258
258
Employee Age
21 19 63 31 27 56 45 41 19
31 41 23 01 14 54 03 12 33
Smith Regan Fox
L A S T N A M E
EMPLOYEE #
09/02/2012
259
259
OLAP Features
09/02/2012
260
260

Sales Volumes
M O D E L
Mini Van Coupe
6 3 4
Blue
5 5 3
Red
4 5 2
White
o
Blue Red White
6 5 4
3 5 5
MODEL
4 3 2
Sedan
Sedan
Mini Van Coupe
COLOR
View #1
View #2
09/02/2012
261

261

Sales Volumes
M O D E L
C O L O R
Blue
C O L O R
Blue
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
( ROTATE 90 )
DEALERSHIP
DEALERSHIP
MODEL
View #1
View #2
View #3
M O D E L
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
MODEL
COLOR
COLOR
View #4
View #5
View #6
09/02/2012
262

262

Sales Volumes
M O D E L
Mini Van
Coupe
Carr Clyde
Carr Clyde
Normal Blue
Metal Blue
DEALERSHIP
COLOR
09/02/2012
263
263
DISTRICT DEALERSHIP
09/02/2012
264
264

East West Central
09/02/2012
265
265

90 80 70 60 50 Inflows ( $M) 40 30 20 10 0

09/02/2012
266
266

267


HOLAP - Hybrid OLAP

DOLAP - Desk OLAP

09/02/2012
268
268
OLAP
Cube
Web Browser
OLAP Tools
OLAP Applications
09/02/2012
269
269
MOLAP - Features
09/02/2012
270
270
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
271
271
ROLAP - Features
09/02/2012
272
272

OLAP Cube
Any Client
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
273
273
HOLAP - Features
09/02/2012
274
274
MOLAP
ROLAP
HOLAP
Slow
Cost
Where to apply?
09/02/2012
275
275
276
09/02/2012
276
09/02/2012
277
277
278

279

280

281

282

283

284
285
286
Unit Testing
Unit Testing
287
Integration Testing
288
Performance Testing

289
Acceptance testing
290
Questions
291
Thank You
292
293
Content
294
Content [contd]
295
An Overview
296

297

298
299
300
Content
301
Content [contd]
302
An Overview
303

304

305
306
307
Content
308
Content [contd]
309
An Overview
310

311

312
313
314

315 222
Data Modeling
316
Data Modeling



317

318 222
Star Schema
Dimension Table
Dimension Table
Fact Table
Dimension Table
319 222
Snowflake Schema
Dimension Table
city
region
regId north south
320
321

322
323
324

325

326
327
ETL Architecture
328
ETL Architecture

Data Extraction:

Data loading
329

330
331
332
ETL Tools
333
Metadata Management
334
What Is Metadata?

335
336

337
338



339

340

341
OLAP
342
Agenda
09/02/2012
343
343

09/02/2012
344
344

345
09/02/2012
345
MDDB Concepts

346
RDBMS v/s MDDB:

COLOR DEALER
Relational DBMS
MDDB
Sales Volumes
M O D E L
DEALERSHIP
COLOR
27 x 4 = 108 cells
347
3 x 3 x 3 = 27 cells

09/02/2012
348
348
Issues with MDDB
Sparsity


09/02/2012
349
349
Employee Age
21 19 63 31 27 56 45 41 19
31 41 23 01 14 54 03 12 33
Smith Regan Fox
L A S T N A M E
EMPLOYEE #
09/02/2012
350
350
OLAP Features
09/02/2012
351
351

Sales Volumes
M O D E L
Mini Van Coupe
6 3 4
Blue
5 5 3
Red
4 5 2
White
o
Blue Red White
6 5 4
3 5 5
MODEL
4 3 2
Sedan
Sedan
Mini Van Coupe
COLOR
View #1
View #2
09/02/2012
352

352

Sales Volumes
M O D E L
C O L O R
Blue
C O L O R
Blue
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
( ROTATE 90 )
DEALERSHIP
DEALERSHIP
MODEL
View #1
View #2
View #3
M O D E L
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
MODEL
COLOR
COLOR
View #4
View #5
View #6
09/02/2012
353

353

Sales Volumes
M O D E L
Mini Van
Coupe
Carr Clyde
Carr Clyde
Normal Blue
Metal Blue
DEALERSHIP
COLOR
09/02/2012
354
354
DISTRICT DEALERSHIP
09/02/2012
355
355

East West Central
09/02/2012
356
356

90 80 70 60 50 Inflows ( $M) 40 30 20 10 0

09/02/2012
357
357

358


HOLAP - Hybrid OLAP

DOLAP - Desk OLAP

09/02/2012
359
359
OLAP
Cube
Web Browser
OLAP Tools
OLAP Applications
09/02/2012
360
360
MOLAP - Features
09/02/2012
361
361
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
362
362
ROLAP - Features
09/02/2012
363
363

OLAP Cube
Any Client
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
364
364
HOLAP - Features
09/02/2012
365
365
MOLAP
ROLAP
HOLAP
Slow
Cost
Where to apply?
09/02/2012
366
366
367
09/02/2012
367
09/02/2012
368
368
369

370

371

372

373

374

375
376
377
Unit Testing
Unit Testing
378
Integration Testing
379
Performance Testing

380
Acceptance testing
381
Questions
382
Thank You
383
384

385 222
Data Modeling
386
Data Modeling



387

388 222
Star Schema
Dimension Table
Dimension Table
Fact Table
Dimension Table
389 222
Snowflake Schema
Dimension Table
city
region
regId north south
390
391

392
393
394

395

396
397
ETL Architecture
398
ETL Architecture

Data Extraction:

Data loading
399

400
401
402
ETL Tools
403
Metadata Management
404
What Is Metadata?

405
406

407
408



409

410
411
Content
412
413
414
415
416
Content
417
418
419
Content
420
Content [contd]
421
An Overview
422

423

424
425
426

427 222
428 222

429 222
Data Modeling
430
Data Modeling



431

432 222

433 222
Data Modeling



434 22 2

435 222
Star Schema
Dimension Table
Dimension Table
Fact Table
Dimension Table
436 222
Snowflake Schema
Dimension Table
city
region
regId north south
437
438

439
440
441

442

443
444
ETL Architecture
445
ETL Architecture

Data Extraction:

Data loading
446

447
448
449
ETL Tools
450
Metadata Management
451
What Is Metadata?

452
453

454
455
456



457

458

459



460

461

462
OLAP
463
Agenda
09/02/2012
464
464

09/02/2012
465
465

466
09/02/2012
466
MDDB Concepts

467
RDBMS v/s MDDB:

COLOR DEALER
Relational DBMS
MDDB
Sales Volumes
M O D E L
DEALERSHIP
COLOR
27 x 4 = 108 cells
468
3 x 3 x 3 = 27 cells

09/02/2012
469
469
Issues with MDDB
Sparsity


09/02/2012
470
470
Employee Age
21 19 63 31 27 56 45 41 19
31 41 23 01 14 54 03 12 33
Smith Regan Fox
L A S T N A M E
EMPLOYEE #
09/02/2012
471
471
OLAP Features
09/02/2012
472
472

Sales Volumes
M O D E L
Mini Van Coupe
6 3 4
Blue
5 5 3
Red
4 5 2
White
o
Blue Red White
6 5 4
3 5 5
MODEL
4 3 2
Sedan
Sedan
Mini Van Coupe
COLOR
View #1
View #2
09/02/2012
473

473

Sales Volumes
M O D E L
C O L O R
Blue
C O L O R
Blue
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
( ROTATE 90 )
DEALERSHIP
DEALERSHIP
MODEL
View #1
View #2
View #3
M O D E L
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
MODEL
COLOR
COLOR
View #4
View #5
View #6
09/02/2012
474

474

Sales Volumes
M O D E L
Mini Van
Coupe
Carr Clyde
Carr Clyde
Normal Blue
Metal Blue
DEALERSHIP
COLOR
09/02/2012
475
475
DISTRICT DEALERSHIP
09/02/2012
476
476

East West Central
09/02/2012
477
477

90 80 70 60 50 Inflows ( $M) 40 30 20 10 0

09/02/2012
478
478

479


HOLAP - Hybrid OLAP

DOLAP - Desk OLAP

09/02/2012
480
480
OLAP
Cube
Web Browser
OLAP Tools
OLAP Applications
09/02/2012
481
481
MOLAP - Features
09/02/2012
482
482
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
483
483
ROLAP - Features
09/02/2012
484
484

OLAP Cube
Any Client
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
485
485
HOLAP - Features
09/02/2012
486
486
MOLAP
ROLAP
HOLAP
Slow
Cost
Where to apply?
09/02/2012
487
487
488
09/02/2012
488
09/02/2012
489
489
490

491

492

493

494

495

496
497
498
Unit Testing
Unit Testing
499
Integration Testing
500
Performance Testing

501
Acceptance testing
502
Questions
503
Thank You
504
505
Content
506
Content [contd]
507
An Overview
508

509

510
511
512
Content
513
Content [contd]
514
An Overview
515

516

517
518
519
Content
520
Content [contd]
521
An Overview
522

523

524
525
526

527 222
Data Modeling
528
Data Modeling



529

530 222
Star Schema
Dimension Table
Dimension Table
Fact Table
Dimension Table
531 222
Snowflake Schema
Dimension Table
city
region
regId north south
532
533

534
535
536

537

538
539
ETL Architecture
540
ETL Architecture

Data Extraction:

Data loading
541

542
543
544
ETL Tools
545
Metadata Management
546
What Is Metadata?

547
548

549
550



551

552

553
OLAP
554
Agenda
09/02/2012
555
555

09/02/2012
556
556

557
09/02/2012
557
MDDB Concepts

558
RDBMS v/s MDDB:

COLOR DEALER
Relational DBMS
MDDB
Sales Volumes
M O D E L
DEALERSHIP
COLOR
27 x 4 = 108 cells
559
3 x 3 x 3 = 27 cells

09/02/2012
560
560
Issues with MDDB
Sparsity


09/02/2012
561
561
Employee Age
21 19 63 31 27 56 45 41 19
31 41 23 01 14 54 03 12 33
Smith Regan Fox
L A S T N A M E
EMPLOYEE #
09/02/2012
562
562
OLAP Features
09/02/2012
563
563

Sales Volumes
M O D E L
Mini Van Coupe
6 3 4
Blue
5 5 3
Red
4 5 2
White
o
Blue Red White
6 5 4
3 5 5
MODEL
4 3 2
Sedan
Sedan
Mini Van Coupe
COLOR
View #1
View #2
09/02/2012
564

564

Sales Volumes
M O D E L
C O L O R
Blue
C O L O R
Blue
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
( ROTATE 90 )
DEALERSHIP
DEALERSHIP
MODEL
View #1
View #2
View #3
M O D E L
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
MODEL
COLOR
COLOR
View #4
View #5
View #6
09/02/2012
565

565

Sales Volumes
M O D E L
Mini Van
Coupe
Carr Clyde
Carr Clyde
Normal Blue
Metal Blue
DEALERSHIP
COLOR
09/02/2012
566
566
DISTRICT DEALERSHIP
09/02/2012
567
567

East West Central
09/02/2012
568
568

90 80 70 60 50 Inflows ( $M) 40 30 20 10 0

09/02/2012
569
569

570


HOLAP - Hybrid OLAP

DOLAP - Desk OLAP

09/02/2012
571
571
OLAP
Cube
Web Browser
OLAP Tools
OLAP Applications
09/02/2012
572
572
MOLAP - Features
09/02/2012
573
573
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
574
574
ROLAP - Features
09/02/2012
575
575

OLAP Cube
Any Client
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
576
576
HOLAP - Features
09/02/2012
577
577
MOLAP
ROLAP
HOLAP
Slow
Cost
Where to apply?
09/02/2012
578
578
579
09/02/2012
579
09/02/2012
580
580
581

582

583

584

585

586

587
588
589
Unit Testing
Unit Testing
590
Integration Testing
591
Performance Testing

592
Acceptance testing
593
Questions
594
Thank You
595
596

597 222
Data Modeling
598
Data Modeling



599

600 222
Star Schema
Dimension Table
Dimension Table
Fact Table
Dimension Table
601 222
Snowflake Schema
Dimension Table
city
region
regId north south
602
603

604
605
606

607

608
609
ETL Architecture
610
ETL Architecture

Data Extraction:

Data loading
611

612
613
614
615
616
Content
617
618
619
Content
620
Content [contd]
621
An Overview
622

623

624
625
626

627 222
628 222

629 222
Data Modeling
630
Data Modeling



631

632 222

633 222
Data Modeling



634 222

635 222
Star Schema
Dimension Table
Dimension Table
Fact Table
Dimension Table
636 22 2
Snowflake Schema
Dimension Table
city
region
regId north south
637
638

639
640
641

642

643
644
ETL Architecture
645
ETL Architecture

Data Extraction:
Integrating dissimilar data types Changing codes Adding a time attribute Summarizing data Calculating derived values Renormalizing data

Data loading
646

647
648
649
ETL Tools
650
Metadata Management
651
What Is Metadata?

652
653

654
655
656



657

658

659



660

661

662
OLAP
663
Agenda
09/02/2012
664
664

09/02/2012
665
665

666
09/02/2012
666
MDDB Concepts

667
RDBMS v/s MDDB:

COLOR DEALER
Relational DBMS
MDDB
Sales Volumes
M O D E L
DEALERSHIP
COLOR
27 x 4 = 108 cells
668
3 x 3 x 3 = 27 cells

09/02/2012
669
669
Issues with MDDB
Sparsity


09/02/2012
670
670
Employee Age
21 19 63 31 27 56 45 41 19
31 41 23 01 14 54 03 12 33
Smith Regan Fox
L A S T N A M E
EMPLOYEE #
09/02/2012
671
671
OLAP Features
09/02/2012
672
672

Sales Volumes
M O D E L
Mini Van Coupe
6 3 4
Blue
5 5 3
Red
4 5 2
White
o
Blue Red White
6 5 4
3 5 5
MODEL
4 3 2
Sedan
Sedan
Mini Van Coupe
COLOR
View #1
View #2
09/02/2012
673

673

Sales Volumes
M O D E L
C O L O R
Blue
C O L O R
Blue
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
( ROTATE 90 )
DEALERSHIP
DEALERSHIP
MODEL
View #1
View #2
View #3
M O D E L
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
MODEL
COLOR
COLOR
View #4
View #5
View #6
09/02/2012
674

674

Sales Volumes
M O D E L
Mini Van
Coupe
Carr Clyde
Carr Clyde
Normal Blue
Metal Blue
DEALERSHIP
COLOR
09/02/2012
675
675
DISTRICT DEALERSHIP
09/02/2012
676
676

East West Central
09/02/2012
677
677

90 80 70 60 50 Inflows ( $M) 40 30 20 10 0

09/02/2012
678
678

679


HOLAP - Hybrid OLAP

DOLAP - Desk OLAP

09/02/2012
680
680
OLAP
Cube
Web Browser
OLAP Tools
OLAP Applications
09/02/2012
681
681
MOLAP - Features
09/02/2012
682
682
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
683
683
ROLAP - Features
09/02/2012
684
684

OLAP Cube
Any Client
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
685
685
HOLAP - Features
09/02/2012
686
686
MOLAP
ROLAP
HOLAP
Slow
Cost
Where to apply?
09/02/2012
687
687
688
09/02/2012
688
09/02/2012
689
689
690

691

692

693

694

695

696
697
698
Unit Testing
Unit Testing
699
Integration Testing
700
Performance Testing

701
Acceptance testing
702
Questions
703
Thank You
704
705
Content
706
Content [contd]
707
An Overview
708

709

710
711
712
Content
713
Content [contd]
714
An Overview
715

716

717
718
719
Content
720
Content [contd]
721
An Overview
722

723

724
725
726

727 222
Data Modeling
728
Data Modeling



729

730 222
Star Schema
Dimension Table
Dimension Table
Fact Table
Dimension Table
731 222
Snowflake Schema
Dimension Table
city
region
regId north south
732
733

734
735
736

737

738
739
ETL Architecture
740
ETL Architecture

Data Extraction:

Data loading
741

742
743
744
ETL Tools
745
Metadata Management
746
What Is Metadata?

747
748

749
750



751

752

753
OLAP
754
Agenda
09/02/2012
755
755

09/02/2012
756
756

757
09/02/2012
757
MDDB Concepts

758
RDBMS v/s MDDB:

COLOR DEALER
Relational DBMS
MDDB
Sales Volumes
M O D E L
DEALERSHIP
COLOR
27 x 4 = 108 cells
759
3 x 3 x 3 = 27 cells

09/02/2012
760
760
Issues with MDDB
Sparsity


09/02/2012
761
761
Employee Age
21 19 63 31 27 56 45 41 19
31 41 23 01 14 54 03 12 33
Smith Regan Fox
L A S T N A M E
EMPLOYEE #
09/02/2012
762
762
OLAP Features
09/02/2012
763
763

Sales Volumes
M O D E L
Mini Van Coupe
6 3 4
Blue
5 5 3
Red
4 5 2
White
o
Blue Red White
6 5 4
3 5 5
MODEL
4 3 2
Sedan
Sedan
Mini Van Coupe
COLOR
View #1
View #2
09/02/2012
764

764

Sales Volumes
M O D E L
C O L O R
Blue
C O L O R
Blue
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
( ROTATE 90 )
DEALERSHIP
DEALERSHIP
MODEL
View #1
View #2
View #3
M O D E L
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
MODEL
COLOR
COLOR
View #4
View #5
View #6
09/02/2012
765

765

Sales Volumes
M O D E L
Mini Van
Coupe
Carr Clyde
Carr Clyde
Normal Blue
Metal Blue
DEALERSHIP
COLOR
09/02/2012
766
766
DISTRICT DEALERSHIP
09/02/2012
767
767

East West Central
09/02/2012
768
768

90 80 70 60 50 Inflows ( $M) 40 30 20 10 0

09/02/2012
769
769

770


HOLAP - Hybrid OLAP

DOLAP - Desk OLAP

09/02/2012
771
771
OLAP
Cube
Web Browser
OLAP Tools
OLAP Applications
09/02/2012
772
772
MOLAP - Features
09/02/2012
773
773
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
774
774
ROLAP - Features
09/02/2012
775
775

OLAP Cube
Any Client
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
776
776
HOLAP - Features
09/02/2012
777
777
MOLAP
ROLAP
HOLAP
Slow
Cost
Where to apply?
09/02/2012
778
778
779
09/02/2012
779
09/02/2012
780
780
781

782

783

784

785

786

787
788
789
Unit Testing
Unit Testing
790
Integration Testing
791
Performance Testing

792
Acceptance testing
793
Questions
794
Thank You
795
796

797 222
Data Modeling
798
Data Modeling



799

800 222
Star Schema
Dimension Table
Dimension Table
Fact Table
Dimension Table
801 222
Snowflake Schema
Dimension Table
city
region
regId north south
802
803

804
805
806

807

808
809
ETL Architecture
810
ETL Architecture

Data Extraction:

Data loading
811

812
813
814
ETL Tools
815
Metadata Management
816
What Is Metadata?

817
818

819
820



821

822
823
Content
824
825
Content
826
Content [contd]
827
An Overview
828

829

830
831
832

833 222
Data Modeling
834
Data Modeling



835

836 222
Star Schema
Dimension Table
Dimension Table
Fact Table
Dimension Table
837 222
Snowflake Schema
Dimension Table
city
region
regId north south
838
839

840
841
842

843

844
845
ETL Architecture
846
ETL Architecture

Data Extraction:

Data loading
847

848
849
850
ETL Tools
851
Metadata Management
852
What Is Metadata?

853
854

855
856



857

858

859
OLAP
860
Agenda
09/02/2012
861
861

09/02/2012
862
862

863
09/02/2012
863
MDDB Concepts

864
RDBMS v/s MDDB:

COLOR DEALER
Relational DBMS
MDDB
Sales Volumes
M O D E L
DEALERSHIP
COLOR
27 x 4 = 108 cells
865
3 x 3 x 3 = 27 cells

09/02/2012
866
866
Issues with MDDB
Sparsity


09/02/2012
867
867
Employee Age
21 19 63 31 27 56 45 41 19
31 41 23 01 14 54 03 12 33
Smith Regan Fox
L A S T N A M E
EMPLOYEE #
09/02/2012
868
868
OLAP Features
09/02/2012
869
869

Sales Volumes
M O D E L
Mini Van Coupe
6 3 4
Blue
5 5 3
Red
4 5 2
White
o
Blue Red White
6 5 4
3 5 5
MODEL
4 3 2
Sedan
Sedan
Mini Van Coupe
COLOR
View #1
View #2
09/02/2012
870

870

Sales Volumes
M O D E L
C O L O R
Blue
C O L O R
Blue
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
( ROTATE 90 )
DEALERSHIP
DEALERSHIP
MODEL
View #1
View #2
View #3
M O D E L
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
MODEL
COLOR
COLOR
View #4
View #5
View #6
09/02/2012
871

871

Sales Volumes
M O D E L
Mini Van
Coupe
Carr Clyde
Carr Clyde
Normal Blue
Metal Blue
DEALERSHIP
COLOR
09/02/2012
872
872
DISTRICT DEALERSHIP
09/02/2012
873
873

East West Central
09/02/2012
874
874

90 80 70 60 50 Inflows ( $M) 40 30 20 10 0

09/02/2012
875
875

876


HOLAP - Hybrid OLAP

DOLAP - Desk OLAP

09/02/2012
877
877
OLAP
Cube
Web Browser
OLAP Tools
OLAP Applications
09/02/2012
878
878
MOLAP - Features
09/02/2012
879
879
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
880
880
ROLAP - Features
09/02/2012
881
881

OLAP Cube
Any Client
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
882
882
HOLAP - Features
09/02/2012
883
883
MOLAP
ROLAP
HOLAP
Slow
Cost
Where to apply?
09/02/2012
884
884
885
09/02/2012
885
09/02/2012
886
886
887

888

889

890

891

892

893
894
895
Unit Testing
Unit Testing
896
Integration Testing
897
Performance Testing

898
Acceptance testing
899
Questions
900
Thank You
901
Content [contd]
902
An Overview
903

904

905
906
907

908 222
Data Modeling
909
Data Modeling



910

911 222
Star Schema
Dimension Table
Dimension Table
Fact Table
Dimension Table
912 222
Snowflake Schema
Dimension Table
city
region
regId north south
913
914

915
916
917

918

919
920
ETL Architecture
921
ETL Architecture

Data Extraction:

Data loading
922

923
924
925
ETL Tools
926
Metadata Management
927
What Is Metadata?

928
929

930
931



932

933

934
OLAP
935
Agenda
09/02/2012
936
936

09/02/2012
937
937

938
09/02/2012
938
MDDB Concepts

939
RDBMS v/s MDDB:

COLOR DEALER
Relational DBMS
MDDB
Sales Volumes
M O D E L
DEALERSHIP
COLOR
27 x 4 = 108 cells
940
3 x 3 x 3 = 27 cells

09/02/2012
941
941
Issues with MDDB
Sparsity


09/02/2012
942
942
Employee Age
21 19 63 31 27 56 45 41 19
31 41 23 01 14 54 03 12 33
Smith Regan Fox
L A S T N A M E
EMPLOYEE #
09/02/2012
943
943
OLAP Features
09/02/2012
944
944

Sales Volumes
M O D E L
Mini Van Coupe
6 3 4
Blue
5 5 3
Red
4 5 2
White
o
Blue Red White
6 5 4
3 5 5
MODEL
4 3 2
Sedan
Sedan
Mini Van Coupe
COLOR
View #1
View #2
09/02/2012
945

945

Sales Volumes
M O D E L
C O L O R
Blue
C O L O R
Blue
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
( ROTATE 90 )
DEALERSHIP
DEALERSHIP
MODEL
View #1
View #2
View #3
M O D E L
COLOR
( ROTATE 90 )
MODEL
( ROTATE 90 )
DEALERSHIP
MODEL
COLOR
COLOR
View #4
View #5
View #6
09/02/2012
946

946

Sales Volumes
M O D E L
Mini Van
Coupe
Carr Clyde
Carr Clyde
Normal Blue
Metal Blue
DEALERSHIP
COLOR
09/02/2012
947
947
DISTRICT DEALERSHIP
09/02/2012
948
948

East West Central
09/02/2012
949
949

90 80 70 60 50 Inflows ( $M) 40 30 20 10 0

09/02/2012
950
950

951


HOLAP - Hybrid OLAP

DOLAP - Desk OLAP

09/02/2012
952
952
OLAP
Cube
Web Browser
OLAP Tools
OLAP Applications
09/02/2012
953
953
MOLAP - Features
09/02/2012
954
954
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
955
955
ROLAP - Features
09/02/2012
956
956

OLAP Cube
Any Client
Relational DW
Web Browser
SQL
OLAP Tools
OLAP Applications
09/02/2012
957
957
HOLAP - Features
09/02/2012
958
958
MOLAP
ROLAP
HOLAP
Slow
Cost
Where to apply?
09/02/2012
959
959
960
09/02/2012
960
09/02/2012
961
961
962

963

964

965

966

967

968
969

Data Warehouse Concepts: Avinash Kanumuru Diya Jana Debyajit Majumder

Hochgeladen von

Dokumentinformationen

Originalbeschreibung:

Originaltitel

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Data Warehouse Concepts: Avinash Kanumuru Diya Jana Debyajit Majumder

Hochgeladen von

Copyright:

Verfügbare Formate

Data Warehouse Concepts

Avinash Kanumuru Diya Jana Debyajit Majumder

2009 Wipro Ltd - Confidential 2009 Wipro Ltd - Confidential

2009 Wipro Ltd - Confidential 2009 Wipro Ltd - Confidential

2009 Wipro Ltd - Confidential 2009 Wipro Ltd - Confidential