Fast Cross Validation Via Sequential Analysis - Talk

Fast CrossValidation via Sequential Analysis Tammo Krueger Danny Panknin Mikio Braun Motivation Fast CV
Algorithm Meta-parameters Example Run
Fast Cross-Validation via Sequential Analysis

Tammo Krueger Danny Panknin Mikio Braun
Experiments
Test Error Speed Increase
Conclusion
Machine Learning Group Technische Universitaet Berlin
16.12.2011 Big Learning Workshop
Motivation
Cross-validation is an indispensable tool for applied ML but unfortunately very time consuming Example tted Q iteration:
Cross-Validation
10 10 10
parameter fold
50
max. iter.
10 = 500, 000 reg. problems

reps.
Experiments
Conclusion
Directly optimizing the error landscape to avoid calculations dicult due to noise Our approach: use increasing subsets of the training data
1 2 3
smaller subsets less training time more training data better error estimate relative behavior of parameter congurations converges
Motivation Main Idea (Average over 500 Reps.)

qqqqqqqqqqqqq
qqqqq
qqq
qq
qq
qq
q qq q q q q q q q q q qq qq qqq qq q
qqqq
qqqqq
qqq
0.4
Size
q
50 100 250 500 750
Class. Error
0.3
Experiments
0.2
1000
Conclusion
0.1 4 3 2 1 0 1 2
log()
Motivation Main Idea (Individual Reps.)

Fast CrossValidation via Sequential Analysis Tammo Krueger Danny Panknin Mikio Braun Motivation
0.4 1 0.5
q qqqqqqqqqqqqqqqq qqqqqq qqq qq q q q q qq q q qq q q q qq q q qq qq qqq q qqqq qqqq
2
qqqqqqqqqqqqqqqqqq qqqqqq q q qqqq q qq q qq q qq qqq q qq q q qq qq q q qq qqqqqq
3
qq qqqqqqqqqqqqqqq qqq qqqq qqqqqqq q qq qqqq q qq qqq q qq qq q qqq qqqqqqqq
0.4
0.3
0.2
0.1 4 0.5
qq q qqqqqqqqqqqqqqqqqq qqqq qqqqqq q q q qqqq qqq q qq qq q q qq q qqq q qq qqqq
5
qqq qqqqqqqqqqqqqqqq qqqqq q qq qqq qq q q qq qq q q qq qq q q q q qq q qq q qqqqqqq
6
qqqqqqqqqqqqqqq qqqqqqqqqq qq qqqqqqq q qqq q qqq q qq qq q qq q qqq qqqqqqq
Size
q
Class. Error
Fast CV
50 100
0.3
250 500
0.2
750 1000
Experiments
0.1 7 0.5
qqqqqqqqqqqqqqqq q qqqqq qq qqqq q q q qqq qqq q qq q qqq q qqq qqq q q qq qqqqqq
8
qqqqqqqqqqqqqqqqq qqqqq qqqq q qq qqqqqqq qqq qq qq q qq qq q q qq q qq qqqqqq
9
q qqqqqqqqqqqqqq qqqqq qqqq q q q qqq q qq qqq qq q q qq q qq q qqq q qqqq qqqqqqq
Conclusion
0.4
0.3
0.2
0.1 4 3 2 1 0 1 2 4 3 2 1 0 1 2 4 3 2 1 0 1 2
log()
Motivation Exploitation
Observations: Individual runs are noisy, but at least we can see the tendency A lot of underperforming parameter congurations We can estimate the correct parameter on a suciently large subset of the data Exploitation: Transformation of the pointwise test errors of the congurations into a binary top or op scheme Dropping of signicant loser congurations along the way via tests from the sequential analysis framework Early stopping of the procedure, when we have seen enough data for a stable parameter estimation
Experiments
Conclusion
Fast Cross-Validation Procedure Algorithm

20
conf. c1 c2 c3 . . . ck2 ck1 ck
d1 -2.2 -1.9 -1.4
d2 -1.9 -2.4 -0.9 . . .
data points d3 -1.8 -2.3 -0.7
dn1 2.1 1.9 0.5 . . .
dn 1.5 2.4 0.5
op op op top top top
1 0 0 0
2 0 1 1
3 0 0 1
4 0 0 0
steps 5 1 0 0 . . .
6 0 0 1 1 1 1
7 0 0 0 1 0 0
8 0 0 0 0 1 1
9 0 0 0 1 1 1
10 0 0 0 1 1 1
() ()
0.6 0.6 0.7 -0.8 -0.4 0.1 0.5 0.7 -0.9 -0.1 0.5 0.4 0.6 -0.3 0.0 pointwisePerformance matrix
0 0 1
1 0 1 1 1 1 1 1 1 0 1 1 trace matrix
Sa(0, 1, l, l)
c1 ck 15 Cumulative Sum
WINNER
Experiments
c3 . . . ck2 ck1 ck
7 0 1 0 0
8 0 . . . 0 1 1
9 0 1 1 1
10 0 1 1 1
Conclusion
10
1 1 0 1 1 0 0 0 0 0 0 0 0 0 0 5 10 Step
1 0
LOSER
H0(0, 1, l, l)
15 20
= N/20 modelSize = 10 n = N 10
Meta-parameters Selection of Test Parameters .

Fast CrossValidation via Sequential Analysis Tammo Krueger Danny Panknin Mikio Braun
Cumulative Sum
c1 c2 c3 . . .
-2.2 -1.9 -1.4
-1.9 -2.4 -0.9 . . .
-1.8 -2.3 -0.7
2.1 1.9 0.5 . . .
1.5 2.4 0.5
op op op top top top
0 0 0 0 0 1
0 1 1
0 0 1
0 0 0
1 0 0
0 0 1 1 1 1
0 0 0 1 0 0
0 0 0 0 1 1
ck2 ck1 ck
0.6 0.6 0.7 -0.8 -0.4 0.1 0.5 0.7 -0.9 -0.1 0.5 0.4 0.6 -0.3 0.0 pointwisePerformance matrix
. .
1 0 1 1 1 1 1 1 1 0 1 1 trace matrix
20
Sa(0, 1, l, l)
c1 ck
WINNER
15
c3 . . . ck2 ck1 ck
7 0 1 0 0
8 0 . . .
Motivation Fast CV
0 1 1
10
Experiments
1 1 0 0 0 0 0 0 0 0 0 0 5 10 Step
1 0
1 0
LOSER
H0(0, 1, l, l)
15 20
= modelSize = n=
Conclusion
(0 , 1 ) =argmax
0 ,1
H0 (0 , 1 , l , l )
s.t. Sa (0 , 1 , l , l ) (steps 1, steps]
Meta-parameters False Negative Rate

0.8
q q
q q q q
False Negative Rate
0.6
Pi
q
0.1 0.2
0.3 0.4 0.5
0.4
0.2
Experiments
0.0
Change Point
10
15
Conclusion
l log 1l log 1 1 l cp 0 with steps log / log 2 1l 11 steps l log l log 10
security zone (false negative rate of 0)
Fast Cross-Validation Procedure Example Run

600
500
400
Configuration
Status 300 flop top out 200 80 150 100 100 50 150 100 50 60 40 20 70 60 50 40 30 20 10 4 5 6 5 6 7
Experiments
Conclusion
234
345
Step
Experimental Setup
8 classication and 7 regression data sets For each dataset:

for parameter estimation, 1 for test error estimation 2 SVM/SVR and Kernel Ridge Regression/Kernel Logistic Regression with Gaussian kernel using 610 parameter congurations Parameter estimation with:
1 2
Experiments
Full 10-fold cross-validation Fast cross-validation procedure with 10 steps
Conclusion
Repeated 50 times with dierent splits for each dataset Compare:

Test error dierence of fast versus full cross-validation time full Relative speed factor, i.e. time fast cross-validation cross-validation
Experiments Test Error

0.06
q q
Classification
Regression
q q q q q q
0.04
q q
q q
MSE Fast CV MSE Full CV
0.02
q q q q q q q q q q q q q q q q q q q q q q q q q
q q q q q q q q
method kreg sv
q
0.00
q q q
Experiments
q q
0.02
Conclusion
0.04
q q
waveform
ringnorm
twonorm
pumadyn32nm
bank32nm
covtype
german
banana
heavisine
image
kin32nm
splice
doppler
bumps
blocks
Experiments Speed Increase

Classification 140
q
Regression
q q q
120
q
100
Time Full CV / Time Fast CV
q q
q q
80
method kreg sv
60
q q q q q q q q
40
Experiments
20
q q q q q
Conclusion
pumadyn32nm
waveform
ringnorm
twonorm
bank32nm
covtype
german
banana
heavisine
image
kin32nm
splice
doppler
bumps
blocks
Fast Cross-Validation Procedure Summary

Motivation: we can estimate the correct parameter on a suciently large subset of the data Transformation: Race of congurations evaluated on linearly increasing subsets of the data At each step of this race:
Transform the test errors on individual data points of the remaining congurations into a binary top or op scheme 2 Drop signicant loser congurations along the way using tests from the sequential analysis framework 3 Apply distribution free testing techniques to decide, whether we have gathered enough evidence for a stable parameter estimation
1
Experiments
Conclusion
Questions? Remarks? Thanks for your attention!
Experiments
Conclusion
Experiments Traces Classication

banana 600 500 400 300 200 100 0 600 500 400 300 200 100 0 600 500 400 300 200 100 0
q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q qq qq q q q qq q q q qq q q qq q q q q q q q q q q q q q q q q q q q q qq q q q qq q q qq qq q q q q q q q q q q q q q q q q q q q q qq q q q q q q q q q q q q q q q q q q q qq qq q q q qq qq qq q q q q q q qq qq q q q q q q q q q q q q q q q q q qq q q qq q q q q q qq q qq q qq qq q q qq qq q qq q q q q q q q
covtype
german
q q q q q q q q q q q
q q qq q q q q
image
ringnorm
splice
value
Fast CV
method kreg sv
q q q q q q q q q q q q q q qq q q q q q q q qq q qq q qq q qq
Experiments
twonorm
waveform
Conclusion
S1 S2 S3 S4 S5 S6 S7 S8 S9S10
S1 S2 S3 S4 S5 S6 S7 S8 S9S10
S1 S2 S3 S4 S5 S6 S7 S8 S9S10
variable
Experiments Traces Regression

bank32nm 600 500 400 300 200 100 0 600 500 400 300 200 100 0 600 500 400 300 200 100 0
q q q q qq q q q q q qq qq q q q q q q qq qq q q q q qq q q qq q q qq q q q q q q q q q q q q q q q q q q q q q q q q q q qq qq q qq qq
blocks
bumps
doppler
heavisine
q q
kin32nm
value
Fast CV
method kreg sv
Experiments
pumadyn32nm
Conclusion
S1 S2 S3 S4 S5 S6 S7 S8 S9S10
S1 S2 S3 S4 S5 S6 S7 S8 S9S10
S1 S2 S3 S4 S5 S6 S7 S8 S9S10
variable

Fast Cross Validation Via Sequential Analysis - Talk

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

Fast Cross Validation Via Sequential Analysis - Talk

Hochgeladen von

Copyright:

Verfügbare Formate

Fast CrossValidation via Sequential Analysis Tammo Krueger Danny Panknin Mikio Braun Motivation Fast CV

Algorithm Meta-parameters Example Run

Fast Cross-Validation via Sequential Analysis

Machine Learning Group Technische Universitaet Berlin

16.12.2011 Big Learning Workshop

10 = 500, 000 reg. problems

Motivation Main Idea (Average over 500 Reps.)

50 100 250 500 750

Motivation Main Idea (Individual Reps.)

Fast Cross-Validation Procedure Algorithm

conf. c1 c2 c3 . . . ck2 ck1 ck

d1 -2.2 -1.9 -1.4

d2 -1.9 -2.4 -0.9 . . .

data points d3 -1.8 -2.3 -0.7

dn1 2.1 1.9 0.5 . . .

dn 1.5 2.4 0.5

op op op top top top

Meta-parameters Selection of Test Parameters .

-2.2 -1.9 -1.4

-1.9 -2.4 -0.9 . . .

-1.8 -2.3 -0.7

2.1 1.9 0.5 . . .

1.5 2.4 0.5

op op op top top top

s.t. Sa (0 , 1 , l , l ) (steps 1, steps]

Meta-parameters False Negative Rate

False Negative Rate

0.3 0.4 0.5

l log 1l log 1 1 l cp 0 with steps log / log 2 1l 11 steps l log l log 10

security zone (false negative rate of 0)

Fast Cross-Validation Procedure Example Run

8 classication and 7 regression data sets For each dataset:

Full 10-fold cross-validation Fast cross-validation procedure with 10 steps

Repeated 50 times with dierent splits for each dataset Compare:

Experiments Test Error

MSE Fast CV MSE Full CV

Experiments Speed Increase

Time Full CV / Time Fast CV

Fast Cross-Validation Procedure Summary

Questions? Remarks? Thanks for your attention!

Experiments Traces Classication

Experiments Traces Regression

Das könnte Ihnen auch gefallen