Data Preparation

Data Preparation
Dr. Saed Sayad

University of Toronto
2010
saed.sayad@utoronto.ca
1 http://chem-eng.utoronto.ca/~datamining/
Data Mining Steps
1
Problem Definition
2
Data Preparation
3
Data Exploration
4
Modeling
5
Evaluation
6
Deployment
http://chem-eng.utoronto.ca/~datamining/ 2
1. Problem Definition
Understanding the project objectives and
requirements from a business perspective,
and then converting this knowledge into a
data mining problem definition with a
preliminary plan designed to achieve the
objectives.
Source: http://www.crisp-dm.org/Process/index.htm
2- Data Preparation
The data preparation step covers all
activities to construct the final dataset
for modeling from the raw data. Tasks
include database, table, record, and field
selection as well as cleaning, aggregation
and transformation of data.
Data Preparation
Modeling Data
Data
Text
Data
DSN
ETL
Data Sources
Text Files
Relational
Database
Multi-dimensional
Database
Entities File Table Cube
Attributes Row and Col
Record, Field,
Index
Dimension, Level,
Measurement
Methods Read, Write
Select, Insert,
Update,
Delete
Drill down, Drill
up, Drill through
Language - SQL MDX
Data Types
Data
Measurement
Ratio
Interval
Counting
Ordinal
Nominal
Numerical
Categorical
Denormalization
One Row per Subject
Tranformation
Customer
Customer
Transformed
1 to 1
Transaction
Transaction
Transformed
1 to 1
1 to N
1 to N
Copy and Aggregate
Customer
Transaction
Copy Aggregate
Data Preparation - Aggregation
Aggregation
Categorical
Count
Count%
Numeric
Count, Sum
Mean, Std
Min, Max
One to Many Relationship
Customer ID Age Married
1 25 N
2 38 Y
3 46 Y
Transaction ID Customer ID
Purchased
Amount
1 1 250
2 1 125
3 2 100
4 2 85
5 2 24
6 3 400
Customers
Transactions
1
N
Data Preparation - Copy
Transaction ID Customer ID
Purchased
Amount
Age Married
1 1 250 25 N
2 1 125 25 N
3 2 100 38 Y
4 2 85 38 Y
5 2 24 38 Y
6 3 400 46 Y
Data Preparation - Aggregation
Customer ID Age Married
Purchased
Count
Purchased
Total
1 25 N 2 375
2 38 Y 3 209
3 46 Y 1 400
Data Transformation and Cleansing
Variable
Categorical Numeric
Missing Values Missing Values
Invalid Values Invalid & Outliers
Encoding Binning
Missing Values
Education
0
500,000
1,000,000
1,500,000
2,000,000
2,500,000
B
L
A
N
K 1 2 3 4
F
r
e
q
u
e
n
c
y
83%
Missing Value
Invalid Values
doc_type_id
0
200,000
400,000
600,000
800,000
1,000,000
1,200,000
1,400,000
N
U
L
L
Z X 1 2 3
F
r
e
q
u
e
n
c
y
Invalid
Missing and Invalid Values and Outliers
Months in Business
Box Plot
Outliers
*
Missing Values
Fill in missing values manually based on our
domain knowledge
Ignore the records with missing data
Fill in it automatically:
A global constant (e.g., ?)
The variable mean
Inference-based methods such as Bayes rule,
decision tree, or EM algorithm
Managing Outliers
Data points inconsistent with the majority of data
Different outliers
Valid: CEOs salary
Noisy: Ones age = 200, widely deviated points
Removal methods
Box plot
Clustering
Curve-fitting
Encoding Categorical Variables
Encoding is the process of transforming
categorical variables into numerical
counterparts.
Encoding methods:
Binary method
Ordinal Method
Target based Encoding
Encoding
Binary method:
for free: 1, 0, 0
own: 0, 1, 0
rent: 0, 0, 1
Ordinal method:
own: 1
for free: 3
rent: 5
Housing (for free, own, rent)
Binning Numerical Variables
Binning is the process of transforming
numerical variables into categorical
counterparts.
Binning methods:
Equal Width
Equal Frequency
Entropy Based
Binning
Variable: 0, 4, 12, 16, 16, 18, 24, 26, 28
Equi-width binning:
Bin 1: 0, 4 [-,10) bin
Bin 2: 12, 16, 16, 18 [10,20) bin
Bin 3: 24, 26, 28 [20,+) bin
Equi-frequency binning :
Bin 1: 0, 4, 12 [-, 14) bin
Bin 2: 16, 16, 18 [14, 21) bin
Bin 3: 24, 26, 28 [21,+) bin
Binning
Months in Business
Summary
In the data preparation step the final modeling
dataset is constructed from the raw data.
One Row per Subject is the heart of the data
preparation activities for building the modeling
dataset.
Tasks include database, table, record, and field
selection as well as cleaning, aggregation and
transformation of data also taking care of missing
values, invalid values and outliers.

Data Preparation

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Data Preparation

Hochgeladen von

Copyright:

Verfügbare Formate

Data Preparation

Dr. Saed Sayad

Das könnte Ihnen auch gefallen