Data Mining & Data Warehousing 1

DATA MINING
&
DATA WAREHOUSING

Submitted By
-K.A.Sha Rajesh Rajan,

-G.Sivakumar,
(DON BOSCO ARTS & SCIENCE),
DHARMAPURI-636 809.

Abstract

In this topic we surveyed the important discipline of data mining, which uses database

technology to discover additional knowledge or patterns inn the data. We gave an illustrative example

of knowledge discovery in database, which had a wider scope, the data mining among the various

techniques, we focused on the details of association rule mining classification and clustering. These

data warehouses provide storage, functionality, and responsiveness to queries beyond the capabilities

of transaction oriented databases.

A data warehouse is a collection of related data and a database system as a database and

database software together. In data warehousing provides data access to the enterprise.
Data Mining & Data Warehousing 2
CONTENTS:
- Overview
- Goals
- Scope
- Fundamental Information Systems
- Architecture
- Working
- Profitable Applications
- Future Developments
- Conclusion

An Introduction to Data Mining: become part of a data warehouse.

The term data mining refers loosely to the

process of semi automatically analyzing large
databases to find useful patterns. The data mining
deals with “Knowledge discovery in databases”. It
is an information extraction activity whose goal is
to discover hidden facts contained in databases.
Typical applications include market segmentation,
customer profiling, fraud detection, evaluation of
retail promotions, and credit risk analysis

An Introduction to Data warehousing:

A data warehouse is a collection of data

gathered and organized so that it can easily by
analyzed, extracted, synthesized, and otherwise be
used for the purposes of further understanding the
data. It may be contrasted

with data that is gathered to meet immediate

business objectives such as order and payment
transactions, although this data would also usually
Data Mining & Data Warehousing 3
• To mix it with information from other,
often external, sources of data.
Overview:
The Scope of Data Warehousing:
Data mining, the extraction of hidden
predictive information from large databases, is a
powerful new technology with great potential to
help companies focus on the most important
information in their data warehouses.

Data Warehousing has grown out of the

repeated attempts on the part of various
researchers and organizations to provide their
organizations flexible, effective and efficient
means of getting at the sets of data that have come
to represent one of the organization's most critical
and valuable assets.

Goal of Data Mining:

• Simplification and automation of the
overall statistical process, from data
source(s) to model application
• Replace statistician ð Better models, less
grunge work
• Statistical expertise required to compare
different techniques
• Build intelligence into the software.
Goal of Data Warehousing:
The goal of data warehousing is :
• Free the information that is locked up in
the operational databases.
To use data models and/or server
technologies that speed up querying and reporting
and that are not appropriate for transaction
processing
To provide a repository of transaction
processing system data that contains data from a
longer span of time than can efficiently be held in
a transaction processing system and/or to be able
to generate reports "as was" as of a previous point
in time.
To prevent persons who only need to
query and report transaction processing
system data from having any access
Data Mining & Data Warehousing
whatsoever to transaction processing
system databases and logic used to
maintain those databases.
An Architecture for Data Mining:
To best apply the advanced techniques,
they must be fully integrated with a data
warehouse as well as flexible interactive business
analysis tools. Many data mining tools currently
operate outside of the warehouse, requiring extra
steps for extracting, importing, and analyzing the
4
data.
The ideal starting point is a data warehouse
containing a combination of internal data tracking
all customer contact coupled with external market
data about competitor activity.
An Architecture for Data Warehousing:
A Data Warehouse Architecture (DWA) is a
way of representing the overall structure of data,
communication, processing and presentation that
exists for end-user computing within the
enterprise. The architecture is made up of a
number of interconnected parts:
Operational Database / External
Database Layer:
Data Mining & Data Warehousing
Operational systems process data to support
critical operational needs. In order to do that,
operational databases have been historically
created to provide an efficient processing
structure for a relatively small number of well-
defined business transactions.
Information Access Layer
The Information Access layer of the Data
Warehouse Architecture is the layer that the end-
user deals with directly. In particular, it represents
the tools that the end-user normally uses day to
day, e.g., Excel, Lotus 1-2-3, Focus, Access, SAS,
etc. This layer also includes the hardware and
software involved in displaying and printing
reports, spreadsheets, graphs and charts for
analysis and Presentation.
Data Access Layer:
5
The Data Access Layer of the Data
Warehouse Architecture is involved with allowing
the Information Access Layer to talk to the
Operational Layer.It is responsible for interfacing
between Information Access tools and
Operational Databases.
Data Directory (Metadata) Layer:
In order to provide for universal data
access, it is absolutely necessary to maintain some
form of data directory or repository of meta-data
information. Meta-data is the data about data
within the enterprise.
In order to have a fully functional
warehouse, it is necessary to have a variety of
meta-data available, data about the end-user views
of data and data about the operational databases.
Ideally, end-users should be able to access data
from the data warehouse (or from the operational
databases) without having to know where that
data resides or the form in which it is stored.
Process Management Layer
The Process Management Layer is involved
in scheduling the various tasks that must be
accomplished to build and maintain the data
warehouse and data directory information. The
Process Management Layer can be thought of as
the scheduler or the high-level job control for the
many processes (procedures) that must occur to
keep the Data Warehouse up-to-date.
Data Mining & Data Warehousing
Application Messaging Layer: T information access data from operational and/or
The Application Message Layer has to do
with transporting information around the
enterprise computing network. Application
Messaging is also referred to as "middleware", but
it can involve more that just networking protocols.
Application Messaging can also be used to collect
transactions or messages and deliver them to a
certain location at a certain time. Application
Messaging in the transport system underlying the
Data Warehouse.
6
external databases.Data Staging may also involve
data quality analysis programs and filters that
identify patterns and data structures within
existing operational data.
Central Data Warehouses
The central data warehouse is a single physical
database that contains all of the data for a specific
functional area, department, division, or
enterprise. Central Data Warehouses are often
selected where there is a common need for
informational data and there are large numbers of
end-users already connected to a central computer
or network.

Data Warehouse (Physical) Layer:

The (core) Data Warehouse is where the

actual data used primarily for informational uses
occurs. In a Physical Data Warehouse, copies, in
some cases many copies, of operational and or
external data are actually stored in a form that is
easy to access and is highly flexible.

Data Staging Layer

The final component of the Data

Warehouse Architecture is Data Staging. Data
Staging is also called copy management or
replication management, but in fact, it includes all
of the processes necessary to select, edit,
summarize, combine and load data warehouse and
Data Mining & Data Warehousing
Distributed Data Warehouses:
Distributed Data Warehouses are just what
their name implies. They are data warehouses in
which the certain components of the data
warehouse are distributed across a number of
different physical databases.
How Data Mining Works:
The technique that is used to perform these
feats in data mining is called modeling.As the
marketing director you have access to a lot of
information about all of your customers: their age,
sex, credit history and long distance calling usage
and you also have a lot of information about your
prospective customers.For instance, a simple
7
model for a telecommunications company might
be:98% of the customers who make more than
$60,000/year spend more than $80/month on long
distanceThis model could then be applied to the
prospect data to try to tell something about the
proprietary information that this
telecommunications company does not currently
have access to. With this model in hand new
customers can be selectively targeted. Test
marketing is an excellent source of data for this
kind of modeling. Table 2 shows another common
scenario for building models: predict what is
going to happen in the future.
Profitable Application:
One of the Important and Current
Application is CREDIT CARD FRAUD
DETECTION & RECOVERY:Online credit card
fraud against merchants can be broken out into
three major categories:
• Organized Fraud
• Opportunistic Fraud
• Cardholder Fraud
Future Developments:Data Warehousing is such a
new field that it is difficult to estimate what new
developments are likely to most affect it. Clearly,
the development of parallel DB servers with
improved query engines is likely to be one of the
most important. Parallel servers will make it
possible to access huge data bases in much less
time.Another new technology is datawarehouses
Data Mining & Data Warehousing 8
that allow for the mixing of traditional numbers,
text and multi-media. The availability of
improved tools for data visualization (business
intelligence) will allow users to see things that
could never be seen before

Conclusion

 The value of warehousing and mining in effective decision making based on concrete evidence from
old data
 Challenges of heterogeneity and scale in warehouse construction and maintenance
 Grades of data analysis tools: straight querying, reporting tools, multidimensional analysis and
mining.