212 Machine Learning Slides

Machine Learning in
Science and Engineering

Gunnar Rtsch
Friedrich Miescher Laboratory
Max Planck Society
Tbingen, Germany
http://www.tuebingen.mpg.de/~raetsch
Gunnar Rtsch
Machine Learning in Science and Engineering
1
CCC Berlin, December 27, 2004
Roadmap
Motivating Examples
Some Background
Boosting & SVMs
Applications
Rationale: Let computers learn to automate

processes and to understand highly
complex data
Gunnar Rtsch
2
Example 1: Spam Classification

From:
smartballlottery@hf-uk.org
Subject: Congratulations
Date: 16. December 2004 02:12:54 MEZ
From:
manfred@cse.ucsc.edu
Subject: ML Positions in Santa Cruz
Date: 4. December 2004 06:00:37 MEZ
LOTTERY COORDINATOR,
INTERNATIONAL PROMOTIONS/PRIZE AWARD DEPARTMENT.
SMARTBALL LOTTERY, UK.
We have a Machine Learning position

at Computer Science Department of
the University of California at Santa Cruz
(at the assistant, associate or full professor level).
DEAR WINNER,
WINNER OF HIGH STAKES DRAWS
Congratulations to you as we bring to your notice, the
results of the the end of year, HIGH STAKES DRAWS of
SMARTBALL LOTTERY UNITED KINGDOM. We are happy to inform you
that you have emerged a winner under the HIGH STAKES DRAWS
SECOND CATEGORY,which is part of our promotional draws. The
draws were held on15th DECEMBER 2004 and results are being
officially announced today. Participants were selected
through a computer ballot system drawn from 30,000
names/email addresses of individuals and companies from
Africa, America, Asia, Australia,Europe, Middle East, and
Oceania as part of our International Promotions Program.
Current faculty members in related areas:

Machine Learning:
DAVID HELMBOLD and MANFRED WARMUTH
Artificial Intelligence:
BOB LEVINSON
DAVID HAUSSLER was one of the main ML researchers in our
department. He now has launched the new Biomolecular Engineering
department at Santa Cruz
There is considerable synergy for Machine Learning at Santa
Cruz:
-New department of Applied Math and Statistics with an emphasis
on Bayesian Methods http://www.ams.ucsc.edu/
-- New department of Biomolecular Engineering
http://www.cbse.ucsc.edu/
Goal: Classify emails into spam / no spam

How? Learn from previously classified emails!
Training:
analyze previous emails
Application: classify new emails
Gunnar Rtsch
3
Example 2: Drug Design
Chemist
F
F
Inactives
F
N N
HO
Cl
HO
HO
N N
HO
N
N
H
O
O
N
S
O
Cl
OH
OH HN
OH
HO
OH
N
Cl
Actives
Gunnar Rtsch
4
The Drug Design Cycle
Chemist
F
F
Inactives
F
N N
HO
Cl
HO
HO
N N
HO
N
N
H
O
O
N
S
O
Cl
OH
OH HN
OH
HO
OH
N
Cl
Actives
former CombiChem
technology
Gunnar Rtsch
5
Learning
F
F
Inactives
N N
HO
Cl
HO
HO
N N
HO
N
N
H
O
O
N
S
O
Cl
OH
OH HN
OH
HO
OH
N
Cl
Machine
Actives
former CombiChem
technology
Gunnar Rtsch
6
Example 3: Face Detection
Gunnar Rtsch
7
Premises for Machine Learning

Supervised Machine Learning
Observe N training examples with label
Learn function
Predict label of unseen example
Examples generated from statistical process

Relationship between features and label
Assumption: unseen examples are generated
from same or similar process
Gunnar Rtsch
8
Problem Formulation
Natural
+1
Plastic
-1
Natural
+1
Plastic
-1
The World:
Data
Unknown Target Function
Unknown Distribution
Objective
Problem:
Gunnar Rtsch
is unknown
9
Problem Formulation
Gunnar Rtsch
10
Example: Natural vs. Plastic Apples
Gunnar Rtsch
11
Gunnar Rtsch
12
Gunnar Rtsch
13
AdaBoost (Freund & Schapire, 1996)

Idea:
Use simple many rules of thumb
Simple hypotheses are not perfect!
Hypotheses combination => increased accuracy
Problems
How to generate different hypotheses?
How to combine them?
Method
Compute distribution
on examples
Find hypothesis on the weighted sample
Combine hypotheses
linearly:
Gunnar Rtsch
14
Boosting: 1st iteration (simple hypothesis)
Gunnar Rtsch
15
Boosting: recompute weighting
Gunnar Rtsch
16
Boosting: 2nd iteration
Gunnar Rtsch
17
Boosting: 2nd hypothesis
Gunnar Rtsch
18
Boosting: recompute weighting
Gunnar Rtsch
19
Boosting: 3rd hypothesis
Gunnar Rtsch
20
Boosting: 4rd hypothesis
Gunnar Rtsch
21
Boosting: combination of hypotheses
Gunnar Rtsch
22
Boosting: decision
Gunnar Rtsch
23
AdaBoost Algorithm
Gunnar Rtsch
24
AdaBoost algorithm
Combination of
Decision stumps/trees
Neural networks
Heuristic rules
Further reading
http://www.boosting.org
http://www.mlss.cc
Gunnar Rtsch
25
property 2
Linear Separation
property 1
Gunnar Rtsch
26
Linear Separation
property 2
property 1
Gunnar Rtsch
27
Linear Separation with Margins
property 2
property 2
property 1
m
ar
gi
n
property 1
large margin => good generalization

Gunnar Rtsch
28
Large Margin Separation

Idea:
Find hyperplane
that maximizes margin
(with
Use
)
for prediction
m
ar
gi
n
Solution:
Linear combination of examples
many s are zero
Support Vector Machines
Demo
Gunnar Rtsch
29
Kernel Trick
Linear in
input space
Gunnar Rtsch
Non-linear in Linear in
input space
feature space
30
Example: Polynomial Kernel
Gunnar Rtsch
31
Support Vector Machines

Demo: Gaussian Kernel
Many other algorithms can use kernels
Many other application specific kernels
Gunnar Rtsch
32
Capabilities of Current Techniques

Theoretically & algorithmically well understood:
Classification with few classes
Regression (real valued)
Novelty Detection
Bottom Line: Machine Learning
works well for relatively simple
objects with simple properties
Current Research
Complex objects
Many classes
Complex learning setup (active learning)
Prediction of complex properties
Gunnar Rtsch
33

Novelty Detection
Current Research
Complex objects
Many classes
Gunnar Rtsch
34

Novelty Detection
Current Research
Complex objects
Many classes
Gunnar Rtsch
35

Novelty Detection
Current Research
Complex objects
Many classes
Gunnar Rtsch
36

Novelty Detection
Current Research
Complex objects
Many classes
Gunnar Rtsch
37
Many Applications
Handwritten Letter/Digit recognition

Face/Object detection in natural scenes
Brain-Computer Interfacing
Gene Finding
Drug Discovery
Intrusion Detection Systems (unsupervised)
Document Classification (by topic, spam mails)
Non-Intrusive Load Monitoring of electric appliances
Company Fraud Detection (Questionaires)
Fake Interviewer identification in social studies
Optimized Disk caching strategies
Optimal Disk-Spin-Down prediction
Gunnar Rtsch
38
MNIST Benchmark
SVM with polynomial kernel

(considers d-th order correlations of pixels)
Gunnar Rtsch
39
MNIST Error Rates
Gunnar Rtsch
40
Face Detection
1.
Classifier
face
non-face
2. Search
2 ( l 1)
600

450

0
.
7

l =1
Gunnar Rtsch
525,820 patches
41
Fast Face Detection

Note: for easy patches, a
quick and inaccurate
classification is sufficient.
Method: sequential
approximation of the classifier
in a Hilbert space
Result: a set of face detection
filters
Romdhani, Blake, Schlkopf, & Torr, 2001

Gunnar Rtsch
42
Example: 1280x1024 Image
1 Filter, 19.8% patches left

Gunnar Rtsch
43
10 Filters, 0.74% Patches left

Gunnar Rtsch
44

Gunnar Rtsch
45

Gunnar Rtsch
46
70 Filters, 0.007 % patches left

Gunnar Rtsch
47
Single Trial Analysis of EEG:towards BCI

Gabriel Curio
Neurophysics Group
Dept. of Neurology
Klinikum Benjamin
Franklin
Freie Universitt
Berlin, Germany
Gunnar Rtsch
Benjamin Blankertz
Klaus-Robert Mller
Intelligent Data Analysis Group, Fraunhofer-FIRST

Berlin, Germany
48
Cerebral Cocktail Party Problem
Gunnar Rtsch
49
The Cocktail Party Problem
How to decompose superimposed signals?

Analogous Signal Processing problem as for cocktail party problem
Gunnar Rtsch
50
The Cocktail Party Problem

input: 3 mixed signals
algorithm: enforce independence
(independent component analysis)
via temporal de-correlation
output: 3 separated signals
"Imagine that you are on the edge of a lake and a friend challenges you to play a game. The game
is this: Your friend digs two narrow channels up from the side of the lake []. Halfway up each one,
your friend stretches a handkerchief and fastens it to the sides of the channel. As waves reach the
side of the lake they travel up the channels and cause the two handkerchiefs to go into motion. You
are allowed to look only at the handkerchiefs and from their motions to answer a series of
questions: How many boats are there on the lake and where are they? Which is the most powerful
one? Which one is closer? Is the wind blowing? (Auditory Scene Analysis, A. Bregman )
(Demo: Andreas Ziehe, Fraunhofer FIRST, Berlin)

Gunnar Rtsch
51
Minimal Electrode Configuration

coverage: bilateral primary
sensorimotor cortices
27 scalp electrodes
reference: nose
bandpass: 0.05 Hz - 200 Hz
ADC 1 kHz
downsampling to 100 Hz
EMG (forearms bilaterally):
m. flexor digitorum
EOG
event channel:
keystroke timing
(ms precision)
Gunnar Rtsch
52
Single Trial vs. Averaging

[V]
15
10
5
0
-5
-10
-15
LEFT
hand
(ch. C4)
15
10
5
0
-5
-10
-15
-600 -500 -400 -300 -200 -100
[V]
RIGHT
hand
(ch. C3)
15
10
5
0
-5
-10
-15
0 [ms]
15
10
5
0
-5
-10
-15
-500 -400 -300 -200 -100
Gunnar Rtsch
-600 -500 -400 -300 -200 -100
0 [ms]
0 [ms]
-500 -400 -300 -200 -100
0 [ms]
53
BCI Setup
ACQUISITION modes:
- few single electrodes
- 32-128 channel electrode caps
- subdural macroelectrodes
- intracortical multi-single-units
EEG parameters:
- slow cortical potentials
- /_ amplitude modulations
- Bereitschafts-/motor-potential
TASK alternatives:
- feedback control
- imagined movements
- movement (preparation)
- mental state diversity
Gunnar Rtsch
54
Gunnar Rtsch
55
Finding Genes on Genomic DNA

Splice Sites: on the boundary
Exons (may code for protein)
Introns (noncoding)
Coding region starts with Translation Initiation Site (TIS: ATG)
Gunnar Rtsch
56
Application: TIS Finding

Engineering Support Vector Machine (SVM) Kernels
That Recognize Translation Initiation Sites (TIS)
GMD.SCAI
Institute for Algorithms

and Scientific Computing
Gunnar Rtsch
Alexander Zien
Thomas Lengauer
GMD.FIRST
Institute for
Computer Architecture
and Software Technology
Gunnar Rtsch
Sebastian Mika
Bernhard Schlkopf
Klaus-Robert Mller
57
TIS Finding: Classification Problem
Select candidate positions

for TIS by looking for ATG
A (1,0,0,0,0)
C (0,1,0,0,0)
G (0,0,1,0,0)
T (0,0,0,1,0)
N (0,0,0,0,1)
Build fixed-length sequence

representation of candidates
(...,0,1,0,0,0,0,...)
Transform sequence into

representaion in real space
1000-dimensional real space

Gunnar Rtsch
58
2-class Splice Site Detection

Window of 150nt
around known splice sites
Positive examples: fixed window around a true splice site

Negative examples: generated by shifting the window
Design of new Support Vector Kernel
Gunnar Rtsch
59
Learning
F
F
Inactives
N N
HO
Cl
HO
HO
N N
HO
N
N
H
O
O
N
S
O
Cl
OH
OH HN
OH
HO
OH
N
Cl
Machine
Actives
former CombiChem
technology
Gunnar Rtsch
60
Three types of Compounds/Points
few
Gunnar Rtsch
actives
more
inactives
plenty
untested
61
Shape/Feature Descriptor
bit =
Shape
Feature type
Feature location
bit number
254230
Shape/Feature Signature
Shape i
~105 bits
Shape j
S. Putta, A Novel Shape/Feature Descriptor, 2001

Gunnar Rtsch
62
Maximizing the Number of Hits
Total number of
active examples
selected
after each batch
On Thrombin
dataset
Largest Selection Strategy
Gunnar Rtsch
63
Concluding Remarks
Computational Challenges
Algorithms can work with 100.000s of examples
(need
operations
Usually model parameters to be tuned
(cross-validation computationally expensive)
Need computer clusters and
Job scheduling systems (pbs, gridengine)
Often use MATLAB
(to be replaced by python: help!)
Machine learning is an exciting research area

involving Computer Science, Statistics & Mathematics
with
a large number of present and future applications (in all situations
where data is available, but explicit knowledge is scarce)
an elegant underlying theory
and an abundance of questions to study.
New computational biology group in Tbingen: looking for people to hire

Gunnar Rtsch
64
Thanks for Your Attention!

Gunnar Rtsch
http://www.tuebingen.mpg.de/~raetsch
Gunnar.Raetsch@tuebingen.mpg.de
Collegues & Contributors: K. Bennett, G. Dornhege, A.
Jagota, M. Kawanabe, J. Kohlmorgen, S. Lemm, C. Lemmen, P.
Laskov, J. Liao, T. Lengauer, R. Meir, S. Mika, K-R. Mller,
T. Onoda, A. Smola, C. Schfer, B. Schlkopf, R. Sommer, S.
Sonnenburg, J. Srinivasan, K. Tsuda, M. Warmuth, J. Weston,
A. Zien
Special Thanks: Nora Toussaint, Julia Lning, Matthias Noll
Gunnar Rtsch
65

212 Machine Learning Slides

Hochgeladen von

Dokumentinformationen

Copyright

Verfügbare Formate

Dieses Dokument teilen

Dokument teilen oder einbetten

Freigabeoptionen

Stufen Sie dieses Dokument als nützlich ein?

Sind diese Inhalte unangemessen?

Copyright:

Verfügbare Formate

212 Machine Learning Slides

Hochgeladen von

Copyright:

Verfügbare Formate

Machine Learning in

Science and Engineering

Machine Learning in Science and Engineering

Rationale: Let computers learn to automate

Machine Learning in Science and Engineering

Example 1: Spam Classification

We have a Machine Learning position

Current faculty members in related areas:

Goal: Classify emails into spam / no spam

Machine Learning in Science and Engineering

Example 2: Drug Design

Machine Learning in Science and Engineering

The Drug Design Cycle

Machine Learning in Science and Engineering

The Drug Design Cycle

Machine Learning in Science and Engineering

Example 3: Face Detection

Machine Learning in Science and Engineering

Premises for Machine Learning

Examples generated from statistical process

Machine Learning in Science and Engineering

Machine Learning in Science and Engineering

Machine Learning in Science and Engineering

Example: Natural vs. Plastic Apples

Machine Learning in Science and Engineering

Example: Natural vs. Plastic Apples

Machine Learning in Science and Engineering

Example: Natural vs. Plastic Apples

Machine Learning in Science and Engineering

AdaBoost (Freund & Schapire, 1996)

Machine Learning in Science and Engineering

Boosting: 1st iteration (simple hypothesis)

Machine Learning in Science and Engineering

Boosting: recompute weighting

Machine Learning in Science and Engineering

Boosting: 2nd iteration

Machine Learning in Science and Engineering

Boosting: 2nd hypothesis

Machine Learning in Science and Engineering

Boosting: recompute weighting

Machine Learning in Science and Engineering

Boosting: 3rd hypothesis

Machine Learning in Science and Engineering

Boosting: 4rd hypothesis

Machine Learning in Science and Engineering

Boosting: combination of hypotheses

Machine Learning in Science and Engineering

Machine Learning in Science and Engineering

Machine Learning in Science and Engineering

Machine Learning in Science and Engineering

Machine Learning in Science and Engineering

Machine Learning in Science and Engineering

Linear Separation with Margins

large margin => good generalization

Machine Learning in Science and Engineering

Large Margin Separation

Machine Learning in Science and Engineering

Machine Learning in Science and Engineering

Example: Polynomial Kernel

Machine Learning in Science and Engineering

Support Vector Machines

Machine Learning in Science and Engineering

Capabilities of Current Techniques

Machine Learning in Science and Engineering