An Introduction

to
Data Warehousing

Center of Excellence
Data Warehousing
Objectives

 At the end of this lesson, you will know :

– What is Data Warehousing
– The evolution of Data Warehousing
– Need for Data Warehousing
– OLTP Vs Warehouse Applications
– Data marts Vs Data Warehouses
– Operational Data Stores
What is a Data Warehouse ?

A data warehouse is a subject-oriented,

integrated, nonvolatile, time-variant collection
of data in support of management's decisions.
- WH Inmon

WH Inmon - Regarded As Father Of Data Warehousing

Subject-Oriented- Characteristics of a Data Warehouse

Data
Operational
Warehouse

Leads Prospects Customers Products

Quotes Regions Time

Orders

Focus is on Subject Areas rather than Applications

Integrated - Characteristics of a Data Warehouse

Appl A - m,f
Appl B - 1,0 m,f
Appl C - male,female

Appl A - balance dec fixed (13,2)

balance dec
Appl B - balance pic 9(9)V99
fixed (13,2)
Appl C - balance pic S9(7)V99 comp-3

Appl A - bal-on-hand
Appl B - current-balance Current balance
Appl C - cash-on-hand

Appl A - date (julian)

Appl B - date (yymmdd) date (julian)
Appl C - date (absolute)

Integrated View Is The Essence Of A Data Warehouse

Non-volatile - Characteristics of a Data Warehouse

insert change

Operational Data
Warehouse
insert
delete
load
read only
access
replace
change

Data Warehouse Is Relatively Static In Nature

Time Variant - Characteristics of a Data Warehouse

Operational Data
Warehouse

Current Value data Snapshot data

• time horizon : 60-90 days • time horizon : 5-10 years
• key may not have element of time • key has an element of time
• data warehouse stores historical
data

Data Warehouse Typically Spans Across Time

Alternate Definitions

A collection of integrated, subject oriented databases

designed to support the DSS function, where each
unit of data is relevant to some moment of time
- Imhoff
Alternate Definitions

Data Warehouse is a repository of data summarized

or aggregated in simplified form from operational
systems. End user orientated data access and
reporting tools let user get at the data for decision
support - Babcock
Evolution of Data Warehousing
1960 - 1985 : MIS Era

• Unfriendly
• Slow
• Dependent on IS programmers
• Inflexible
• Analysis limited to defined reports
Focus on Reporting
Evolution of Data Warehousing

1985 - 1990 : Querying Era

• Adhoc, unstructured access to corporate data

• SQL as interface not scalable

• Cannot handle complex analysis

Focus on Online Querying

Evolution of Data Warehousing
1990 - 20xx : Analysis Era

• Trend Analysis
• What If ?
• Moving Averages
• Cross Dimensional Comparisons
• Statistical profiles
• Automated pattern and rule discovery
Focus on Online Analysis
Need for Data Warehousing

 Better business intelligence for end-users

 Reduction in time to locate, access, and analyze information

 Consolidation of disparate information sources

 Strategic advantage over competitors

 Faster time-to-market for products and services

 Replacement of older, less-responsive decision support

systems

 Reduction in demand on IS to generate reports

OLTP Systems Vs Data Warehouse
Remember

Between OLTP and Data Warehouse systems

users are different

data content is different,

data structures are different

hardware is different
Understanding The Differences Is The Key
OLTP Vs Warehouse

Operational System Data Warehouse

Transaction Processing Query Processing

Predictable CPU Usage Random CPU Usage

Time Sensitive History Oriented

Operator View Managerial View

Normalized Efficient Denormalized Design for

Design for TP Query Processing
OLTP Vs Warehouse
Operational System Data Warehouse
Designed for Atmocity, Designed for quite or static
Consistency, Isolation and database
Durability
Organized by transactions Organized by subject
(Order, Input, Inventory) (Customer, Product)
Relatively smaller database Large database size
Many concurrent users Relatively few concurrent
users
Volatile Data Non Volatile Data
OLTP Vs Warehouse

Operational System Data Warehouse

Stores all data Stores relevant data

Performance Sensitive Less Sensitive to performance

Not Flexible Flexible

Efficiency Effectiveness
Examples Of Some Applications
Manufacturers
Manufacturers Retailers
Retailers
 Target Marketing
 Market Segmentation
 Budgeting
 Credit Rating Agencies
 Financial Reporting and Consolidation
 Market Basket Analysis - POS Analysis
 Churn Analysis
Customers
Customers
 Profitability Management
 Event tracking
Do we need a separate database ?
 OLTP and data warehousing require two very
differently configured systems
 Isolation of Production System from Business
Intelligence System
 Significant and highly variable resource demands of
the data warehouse
 Cost of disk space no longer a concern
 Production systems not designed for query
processing
Data Marts
 Enterprise wide data warehousing projects have a
very large cycle time
 Getting consensus between multiple parties may
also be difficult
 Departments may not be satisfied with priority
accorded to them
 Sometimes individual departmental needs may be
strong enough to warrant a local implementation
 Application/database distribution is also an
important factor
Data Marts

Subject or Application Oriented Business View of

Warehouse
» Finance, Manufacturing, Sales etc.

» Smaller amount of data used for Analytic Processing

» Frequently implemented using MDDBs

A Logical Subset of The Complete Data Warehouse

Data Warehouse and Data Mart

Data Warehouse Data Marts

Scope  Application Neutral  Specific Application
 Centralized, Shared Requirement
 Cross LOB/enterprise  LOB, department
 Business Process
Oriented
Data  Historical Detailed data  Detailed (some history)
Perspective  Some summary  Summarized

Subjects  Multiple subject areas  Single Partial subject

 Multiple partial subjects
 OLTP snapshots
Data Warehouse and Data Mart

Data Warehouse Data Marts

Data Sources  Many  Few
 Operational/ External  Operational, external
Data data
 OLTP snapshots
Implement  9-18 months for first  4-12 months
stage
Time Frame
 Multiple stage
implementation
Characteristics  Flexible, extensible  Restrictive, non
 Durable/Strategic extensible
 Data orientation  Short life/tactical
 Project Orientation
Warehouse or Mart First ?
Data Warehouse First
Expensive
Large development cycle
Change management is
difficult
Difficult to obtain continuous
corporate support
Technical challenges in
building large databases
Data Mart first
Relatively cheap
Delivered in < 6 months
Easy to manage change
Can lead to independent and
incompatible marts
Cleansing, transformation,
modeling techniques may be
incompatible
Operational Data Store - Definition
Can I see credit
report from
Accounts, Sales Data from multiple
from sources is integrated
marketing and for a subject
open order
report from
order entry for
this customer A subject oriented, integrated,
volatile, current valued data store
containing only corporate
Identical queries may
give different results
detailed data Data stored only for
current period. Old
at different times.
Data is either
Supports analysis
archived or moved to
requiring current
Data Warehouse
data
Operational Data Store

 Increasingly becoming integrated with the data

warehouse
 Are nothing but more responsive real time data
warehouses
 Data Mining has anyway forced Data Warehouses
to store transactional level data
Different kinds of Information Needs

Is this medicine available

 Current
Current in stock

What are the tests this

patient has completed so
 Recent
Recent far

Has the incidence of

Tuberculosis increased in
 Historical
Historical last 5 years in Southern
region
OLTP Vs ODS Vs DWH

Characteristic OLTP ODS Data Warehouse

Audience Operating Analysts Managers and
Personnel analysts
Data access Individual records, Individual records, Set of records,
transaction driven transaction or analysis driven
analysis driven
Data content Current, real-time Current and near- Historical
current
Data granularity Detailed Detailed and lightly Summarized and
summarized derived
Data organization Functional Subject-oriented Subject-oriented
Data quality All application All integrated data Data relevant to
specific detailed needed to support a management
data needed to business activity information needs
support a business
activity
OLTP Vs ODS Vs DWH

Characteristic OLTP ODS Data Warehouse

Data redundancy Non-redundant Somewhat Managed
within system; redundant with redundancy
Unmanaged operational
redundancy among databases
systems
Data stability Dynamic Somewhat dynamic Static
Data update Field by field Field by field Controlled batch
Data usage Highly structured, Somewhat Highly
repetitive structured, some unstructured,
analytical heuristic or
analytical
Database size Moderate Moderate Large to very large
Database Stable Somewhat stable Dynamic
structure stability
OLTP Vs ODS Vs DWH

Characteristic OLTP ODS Data Warehouse

Development Requirements Data driven, Data driven,
methodology driven, structured somewhat evolutionary
evolutionary
Operational Performance and Availability Access flexibility
priorities availability and end user
autonomy
Philosophy Support day-to- Support day-to-day Support managing
day operation decisions & the enterprise
operational
activities
Predictability Stable Mostly stable, some Unpredictable
unpredictability
Response time Sub-second Seconds to minutes Seconds to minutes
Return set Small amount of Small to medium Small to large
data amount of data amount of data