Topic 1.

Need for Data Warehousing Overview & Concepts

Subject Incharge : Pratidnya S. Hegde Patil

Introduction to Data Warehouse

Textbook : Data Warehousing Fundamentals, A comprehensive guide for IT Professionals, by Paulraj Ponniah, Publisher: John Wiley & Sons, 2nd Edition

Objectives
Understand the desperate need for strategic information Recognize the information crisis at every enterprise Distinguish between operational and informational systems Learn why past attempts to provide strategic information failed Clearly see why data warehousing is the viable solution

Data and Information

Were told we live in the information age. People often talk about data and information as if there were the same. They are, in many regards, opposite. A datum is just a fact : your name is a fact, your phone number is a fact. Information is data that is presented in a meaningful, understandable and beneficial format. Information is data that has been organized, sequenced, correlated and summarized, such as a phone book.

Data and Information

A phone book is information. It not only contains names and phone numbers, but it correctly associates each persons phone number with their names. It presents this list of correlated names and phone numbers in alphabetical sequence, so that we find the phone number from the name. In addition, it divides the phone numbers into two types; personal and business. It is the function of the computer to convert data to information.

Definitions
Database: The database is a place where you put your data; data that you wish to convert to information at some future time. Database Management System: A DBMS is the software that converts the data in your database to information. It is the DBMS that provides you the capability for cross-referencing, correlating, sorting, summarizing, etc.

Information as A Competitive Weapon

Information technology and quality information are not the goals, but merely to support organizations to reach goals of Superior products and services Greater productivity Eventually success

The Information Crisis

Integrated: Must have a single, enterprise-wide view. Data Integrity: Information must be accurate and must conform to business rules. Accessible: Easily accessible with intuitive access paths, and responsive for analysis. Credible: Every business factor must have one and one value. Timely: Information must be available within the stipulated time frame.

ERP (Enterprise Resource Planning)

A software solution that addresses enterprise needs taking the process view of an organization to meet the organization goals. It integrates all the departments and functions across a company into a single computer system that can serve all those different departments particular needs. It is a single application that supports (manages) all aspects (domains) of a company. It supports the day to day operations of the company. To ensure that the transactions are fast it maintains only the recent data.

Where was ERP lacking?

Thousands of relational database tables, designed and normalized for running the business operations were not at all suitable for providing strategic information. ERP data repositories lacked data from external sources and from other operational systems in the company.

Decision Support System

DSSs Inability to provide strategic information

IT receives too many ad hoc requests, resulting in a large overload. With limited resources, IT is unable to respond to the numerous requests in a timely fashion. Request keep changing all the time. The users require more reports to expand and understand the earlier reports. Users go into a Spiral of asking more, therefore increasing IT load. Users have to depend on IT to provide information. Not usercentric. IT unable to provide a flexible and conducive environment for strategic decision making.

Operational System

Informational Systems

Operational vs DSS

The Evolution of Data Warehousing

Since 1970s, organizations gained competitive

advantage through systems that automate business processes to offer more efficient and cost-effective services to the customer.

This resulted in accumulation of growing amounts of data in operational databases.

The Evolution of Data Warehousing

Organizations now focus on ways to use operational data to support decision-making, as a means of gaining competitive advantage. However, operational systems were never designed to support such business activities. Involved with day to day transactions only. Businesses typically have numerous operational systems with overlapping and sometimes contradictory definitions.

The Evolution of Data Warehousing

Organizations need to turn their archives of data into a source of knowledge, so that a single integrated or consolidated view of the organizations data is presented to the user. A data warehouse was deemed the solution to meet the requirements of a system capable of supporting decision-making, receiving data from multiple operational data sources.

Objectives of Todays Businesses

Access and combine data from a variety of data stores Perform complex data analysis across these data stores Create multidimensional views of data and its metadata Easily summarize and roll up the information across subject areas and business dimensions

These objectives cannot be met easily

Data is scattered in many types of incompatible structures. Lack of documentation has prevented from integration older legacy systems with newer systems Internet software like searching engine needs to be improved Accurate and accessible metadata across multiple organizations is hard to get

A New Type of System Environment : DW

Data is designed for analytical tasks Data from multiple applications Easy to use and conductive to long interactive sessions by users Read-intensive data usage Direct interaction with the system by the users without IT assistance Content updated periodically and stable Content to include current and historical data Ability for users to run queries and get results online Ability for users to initiate reports

Four Levels of Analytical Processing

In modern organization, at least four levels of analytical processing should be supported by information systems First level: Consists of simple queries and reports against current and historical data Second level: Goes deeper and requires the ability to do what if processing across data store dimensions

Four Levels of Analytical Processing

Third level: Needs to step back and analyze what has previously occurred to bring about the current status of the data Fourth level: Analyzes what has happened in the past and what needs to be done in the future in order to bring some specific change

Business Intelligence at the DW

What is a Data Warehouse?

Data Warehousing is a decision support system. It extracts data from various source systems eg : ERP, CRM. It has historical data kept in a single uniform format. So summarizing, A DW is : An ideal environment for data analysis and decision support. Flexible and interactive. 100% user-driven. Very responsive and conducive to the ask-answer-askagain pattern. Provides the ability to discover answers to complex, unpredictable questions.

Characteristics
1.

The new concept is not to generate fresh data, but to make use of the large volumes of existing data and to transform it into forms suitable for providing strategic information. It is an user-centric environment not a product. A computing environment where users can find strategic information. A central database that is loaded from multiple operational databases for the purpose of end-user access and decision support. A data warehouse differs from an operational system in that the data it contains is normally static and updated in a scheduled manner through massive loading procedures. A data warehouse is developed to accommodate random, ad hoc queries and to allow users to drill down to minute levels of detail.

2.

3.

4.

Concept of Data Warehousing

Take all the data from the operational systems. Where necessary, include relevant data from outside, such as industry benchmark indicators. Integrate all the data from the various sources. Remove inconsistencies and transform the data. Store the data in formats suitable for easy access for decision making. This simple concept, involves different functions : data extraction, loading the data, transformation, storage, providing user interfaces.

Blend of Technologies
Different technologies needed to support data warehousing functions.