Beruflich Dokumente
Kultur Dokumente
Course Syllabus
Meeting Days: Tuesday, 4:40P - 7:10P, TL302 Instructor: Slobodan Vucetic, 304 Wachman Hall, vucetic@ist.temple.edu, phone: 204-5535, www.ist.temple.edu/~vucetic Office Hours: Tuesday 2:00 pm - 3:00 pm; Friday 3:00-4:00 pm; or by appointment. Objective:
The course is devoted to information system environments enabling efficient indexing and advanced analyses of current and historical data for strategic use in decision making. Data management will be discussed in the content of data warehouses/data marts; Internet databases; Geographic Information Systems, mobile databases, temporal and sequence databases. Constructs aimed at an efficient online analytic processing (OLAP) and these developed for nontrivial exploratory analysis of current and historical data at such data sources will be discussed in details. The theory will be complemented by hands-on applied studies on problems in financial engineering, e-commerce, geosciences, bioinformatics and elsewhere.
Prerequisites:
CIS 511 and an undergraduate course in databases.
Course Syllabus
Textbook:
(required) J. Han, M. Kamber, Data Mining: Concepts and Techniques, 2001. Additional papers and handouts relevant to presented topics will be distributed as needed.
Topics:
Overview of data warehousing and mining Data warehouse and OLAP technology for data mining Data preprocessing Mining association rules Classification and prediction Cluster analysis Mining complex types of data
Grading:
(30%) Homework Assignments (programming assignments, problems sets, reading assignments); (15%) Quizzes; (15%) Class Presentation (30 minute presentation of a research topic; during November); (20%) Individual Project (proposals due first week of November; written reports due the last day of the finals); (20%) Final Exam.
Course Syllabus
Late Policy and Academic Honesty:
The projects and homework assignments are due in class, on the specified due date. NO LATE SUBMISSIONS will be accepted. For fairness, this policy will be strictly enforced. Academic honesty is taken seriously. You must write up your own solutions and code. For homework problems or projects you are allowed to discuss the problems or assignments verbally with other class members. You MUST acknowledge the people with whom you discussed your work. Any other sources (e.g. Internet, research papers, books) used for solutions and code MUST also be acknowledged. In case of doubt PLEASE contact the instructor.
Motivation:
Necessity is the Mother of Invention
Data explosion problem Automated data collection tools and mature database technology lead to tremendous amounts of data stored in databases, data warehouses and other information repositories We are drowning in data, but starving for knowledge! Solution: Data warehousing and data mining Data warehousing and on-line analytical processing Extraction of interesting knowledge (rules, regularities, patterns, constraints) from data in large databases
Computers have become cheaper and more powerful Competitive Pressure is Strong
Provide better, customized services for an edge (e.g. in Customer Relationship Management)
Traditional techniques infeasible for raw data Data mining may help scientists
in classifying and segmenting data in Hypothesis Formation
Mining?
Look up phone
number in phone directory
Query a Web
search engine for information about Amazon
Description Tasks
Find human-interpretable patterns that describe the data.
Classification: Definition
Given a collection of records (training set )
Each record contains a set of attributes, one of the attributes is the class.
Find a model for class attribute as a function of the values of other attributes. Goal: previously unseen records should be assigned a class as accurately as possible.
A test set is used to determine the accuracy of the model. Usually, the given data set is divided into training and test sets, with training set used to build the model and test set used to validate it.
Classification Example
No Yes No Yes
Test Set
Training Set
Learn Classifier
Model
Classification: Application 1
Direct Marketing
Goal: Reduce cost of mailing by targeting a set of consumers likely to buy a new cell-phone product. Approach:
Use the data for a similar product introduced before. We know which customers decided to buy and which decided otherwise. This {buy, dont buy} decision forms the class attribute. Collect various demographic, lifestyle, and companyinteraction related information about all such customers.
Type of business, where they stay, how much they earn, etc.
Classification: Application 2
Fraud Detection
Goal: Predict fraudulent cases in credit card transactions. Approach:
Use credit card transactions and the information on its account-holder as attributes.
When does a customer buy, what does he buy, how often he pays on time, etc
Label past transactions as fraud or fair transactions. This forms the class attribute. Learn a model for the class of the transactions. Use this model to detect fraud by observing credit card transactions on an account.
Classification: Application 3
Customer Attrition/Churn:
Goal: To predict whether a customer is likely to be lost to a competitor. Approach:
Use detailed record of transactions with each of the past and present customers, to find attributes.
How often the customer calls, where he calls, what time-of-the day he calls most, his financial status, marital status, etc.
Classification: Application 4
Sky Survey Cataloging
Goal: To predict class (star or galaxy) of sky objects, especially visually faint ones, based on the telescopic survey images (from Palomar Observatory).
3000 images with 23,040 x 23,040 pixels per image.
Approach:
Segment the image. Measure image attributes (features) - 40 of them per object. Model the class based on these features. Success Story: Could find 16 new high red-shift quasars, some of the farthest objects that are difficult to find!
Classifying Galaxies
Early Class:
Stages of Formation
Attributes:
Image features, Characteristics of light waves received, etc.
Intermediate
Late
Data Size:
72 million stars, 20 million galaxies Object Catalog: 9 GB Image Database: 150 GB
Clustering Definition
Given a set of data points, each having a set of attributes, and a similarity measure among them, find clusters such that
Data points in one cluster are more similar to one another. Data points in separate clusters are less similar to one another.
Similarity Measures:
Euclidean Distance if attributes are continuous. Other Problem-specific Measures.
Illustrating Clustering
Euclidean Distance Based Clustering in 3-D space.
Clustering: Application 1
Market Segmentation:
Goal: subdivide a market into distinct subsets of customers where any subset may conceivably be selected as a market target to be reached with a distinct marketing mix. Approach:
Collect different attributes of customers based on their geographical and lifestyle related information. Find clusters of similar customers. Measure the clustering quality by observing buying patterns of customers in same cluster vs. those from different clusters.
Clustering: Application 2
Document Clustering:
Goal: To find groups of documents that are similar to each other based on the important terms appearing in them. Approach: To identify frequently occurring terms in each document. Form a similarity measure based on the frequencies of different terms. Use it to cluster. Gain: Information Retrieval can utilize the clusters to relate a new document or search term to clustered documents.
TID
Items
1 2 3 4 5
Bread, Coke, Milk Beer, Bread Beer, Coke, Diaper, Milk Beer, Bread, Diaper, Milk Coke, Diaper, Milk
Rules Discovered:
{Milk} --> {Coke} {Diaper, Milk} --> {Beer}
Regression
Predict a value of a given continuous valued variable based on the values of other variables, assuming a linear or nonlinear model of dependency. Greatly studied in statistics, neural network fields. Examples:
Predicting sales amounts of new product based on advetising expenditure. Predicting wind velocities as a function of temperature, humidity, air pressure, etc. Time series prediction of stock market indices.
Deviation/Anomaly Detection
Detect significant deviations from normal behavior Applications:
Credit Card Fraud Detection
Exploratory-based method:
Try to make sense of a bunch of data without an a priori hypothesis! The only prevention against false results is significance:
ensure statistical significance (using train and test etc.) ensure domain significance (i.e., make sure that the results make sense to a domain expert)
Data Miner:
Notices that the yield is somewhat higher under trees where birds roost. Conclusion: droppings increase yield. Alternative conclusion: moderate amount of shade increases yield.(Identification Problem)
Data Warehouse
Data Cleaning Data Integration
Databases
Creating a target data set: data selection Data cleaning and preprocessing: (may take 60% of effort!) Data reduction and transformation:
Find useful features, dimensionality/variable reduction, invariant representation.
Choosing the mining algorithm(s) Data mining: search for patterns of interest Pattern evaluation and knowledge presentation
visualization, transformation, removing redundant patterns, etc.
Making Decisions Data Presentation Visualization Techniques Data Mining Information Discovery Data Exploration Statistical Analysis, Querying and Reporting
Data Warehouses / Data Marts OLAP, MDA Data Sources Paper, Files, Information Providers, Database Systems, OLTP
DBA
Machine Learning
Data Mining
Visualization
Information Science
Other Disciplines
Data Mining:
Large Data sets Efficiency of Algorithms is important Scalability of Algorithms is important Real World Data Lots of Missing Values Pre-existing data - not user generated Data not static - prone to updates Efficient methods for data retrieval available for use
OLAP: On-line Analytical Processing Multi-Dimensional Data Model (Data Cube) Operations:
Roll-up Drill-down Slice and dice Rotate
An OLAM Architecture
Mining query
User GUI API
Mining result
OLAM Engine
Data Cube API
OLAP Engine
Layer2
MDDB
Meta Data
Filtering&Integration
MDDB
Database API
Data cleaning
Filtering
OLAP
Summaries, trends and forecasts Analysis Multidimensional data modeling, Aggregation, Statistics What is the average income of mutual fund buyers by region by year?
Data Mining
Knowledge discovery of hidden patterns and insights Insight and Prediction Induction (Build the model, apply it to new data, get the result) Who will buy a mutual fund in the next 6 months and why?
Type of result
Method
Example question
Integration of the discovered knowledge with existing knowledge: A knowledge fusion problem Protection of data security, integrity, and privacy