Combining Inductive and Analytical

Learning

[Read Ch. 12]

[Suggested exercises: 12.1, 12.2, 12.6, 12.7, 12.8]
 Why combine inductive and analytical learning?
 KBANN: Prior knowledge to initialize the
hypothesis
 TangetProp, EBNN: Prior knowledge alters
search objective
 FOCL: Prior knowledge alters search operators

1 lecture slides for textbook Machine Learning,

c T. Mitchell, McGraw Hill, 1997
 Inductive and Analytical Learning

Inductive learning Analytical learning

Hypothesis ts data Hypothesis ts domain theo
Statistical inference Deductive inference
Requires little prior knowledge Learns from scarce data
Syntactic inductive bias Bias is domain theory

2 lecture slides for textbook Machine Learning,

c T. Mitchell, McGraw Hill, 1997
 What We Would Like

Inductive learning Analytical learning

Plentiful data Perfect prior knowledge
No prior knowledge Scarce data

General purpose learning method:

 No domain theory ! learn as well as inductive
methods
 Perfect domain theory ! learn as well as
Prolog-EBG

 Accomodate arbitrary and unknown errors in

domain theory
 Accomodate arbitrary and unknown errors in
training data

3 lecture slides for textbook Machine Learning,

c T. Mitchell, McGraw Hill, 1997
Domain theory:
Cup Stable, Liftable, OpenVessel
Stable BottomIsFlat
Liftable Graspable, Light
Graspable HasHandle
OpenVessel HasConcavity, ConcavityPointsUp
Training examples:

p Cups
p p p p pNon-Cups
p p
BottomIsFlat p pppp pp
ConcavityPoints Up p p p p
Expensive p p pp p p
Fragile p p
HandleOnTop p p p
HandleOnSide p pppp ppp p
HasConcavity p pp p p
HasHandle p pppppp p
Light p p pp
MadeOfCeramic p p
MadeOfPaper pp p p
MadeOfStyrofoam

4 lecture slides for textbook Machine Learning,

c T. Mitchell, McGraw Hill, 1997
 KBANN

KBANN (data D, domain theory B )

1. Create a feedforward network h equivalent to B
2. Use Backprop to tune h to t D

5 lecture slides for textbook Machine Learning,

c T. Mitchell, McGraw Hill, 1997
 Neural Net Equivalent to Domain Theory

Expensive
BottomIsFlat Stable
MadeOfCeramic
MadeOfStyrofoam
MadeOfPaper
HasHandle Graspable Liftable Cup
HandleOnTop
HandleOnSide
Light
OpenVessel
HasConcavity
ConcavityPointsUp
Fragile

6 lecture slides for textbook Machine Learning,

c T. Mitchell, McGraw Hill, 1997
 Creating Network Equivalent to Do-
main Theory

Create one unit per horn clause rule (i.e., an AND

unit)
 Connect unit inputs to corresponding clause
antecedents
 For each non-negated antecedent, corresponding
input weight w W , where W is some constant
 For each negated antecedent, input weight
w ?W
 Threshold weight w 0 ?(n ? :5)W , where n is
number of non-negated antecedents
Finally, add many additional connections with
near-zero weights
Liftable Graspable; :Heavy

7 lecture slides for textbook Machine Learning,

c T. Mitchell, McGraw Hill, 1997
 Result of rening the network

Expensive
BottomIsFlat Stable
MadeOfCeramic
MadeOfStyrofoam
MadeOfPaper
HasHandle Graspable Liftable Cup
HandleOnTop
HandleOnSide
Light
Open-Vessel
HasConcavity
ConcavityPointsUp
Fragile Large positive weight
Large negative weight
Negligible weight

8 lecture slides for textbook Machine Learning,

c T. Mitchell, McGraw Hill, 1997
 KBANN Results

Classifying promoter regions in DNA

leave one out testing:
 Backpropagation: error rate 8/106
 KBANN: 4/106
Similar improvements on other classication,
control tasks.

9 lecture slides for textbook Machine Learning,

c T. Mitchell, McGraw Hill, 1997
 Hypothesis space search in KBANN

Hypothesis Space

Hypotheses that
fit training data
equally well
Initial hypothesis
for KBANN

Initial hypothesis
for BACKPROPAGATION

10 lecture slides for textbook Machine Learning,

c T. Mitchell, McGraw Hill, 1997
 EBNN

Key idea:
 Previously learned approximate domain theory
 Domain theory represented by collection of
neural networks
 Learn target function as another neural network

11 lecture slides for textbook Machine Learning,

c T. Mitchell, McGraw Hill, 1997
 Explanation of
training example Stable
in terms of
domain theory:

BottomIsFlat =T
ConcavityPointsUp =T Graspable Liftable Cup
Expensive =T
Fragile =T
HandleOnTop =F
HandleOnSide =T Cup = T
HasConcavity =T
HasHandle =T
Light =T
0.8
MadeOfCeramic =T
MadeOfPaper =F
MadeOfStyrofoam =F

0.2 OpenVessel

Training
derivatives

Target network: BottomIsFlat

ConcavityPointsUp
Expensive Cup
Fragile target
HandleOnTop
HandleOnSide Cup
HasConcavity
HasHandle
Light
MadeOfCeramic
MadeOfPaper
MadeOfStyrofoam

12 lecture slides for textbook Machine Learning,

c T. Mitchell, McGraw Hill, 1997
 Modied Objective for Gradient Descent

2 0 12 3
E= X 666
(f (x ) ? f^(x )) +  2 X BB
B@
@A(x) ? @ f^(x) CC
CA
77
77
i
64 i i i
j @x j
@x j
(x=xi )
5

where
 1? jA(x ) ? f (x )j i i
i
c
 f (x) is target function
 f^(x) is neural net approximation to f (x)
 A(x) is domain theory approximation to f (x)

13 lecture slides for textbook Machine Learning,

c T. Mitchell, McGraw Hill, 1997
 f(x)
h
f(x1)
f(x2)
f(x3) f
g

x1 x2 x3 x x x

14 lecture slides for textbook Machine Learning,

c T. Mitchell, McGraw Hill, 1997
 Hypothesis Space Search in EBNN

Hypothesis Space

Hypotheses that Hypotheses that

maximize fit to maximize fit to data
data and prior
knowledge

TANGENTPROP
Search BACKPROPAGATION
Search

15 lecture slides for textbook Machine Learning,

c T. Mitchell, McGraw Hill, 1997
 Search in FOCL

Cup

Generated by the
domain theory

Cup HasHandle
[2+,3]

Cup HasHandle
[2+,3]
Cup Fragile ...
Cup BottomIsFlat,
[2+,4] Light,
HasConcavity,
ConcavityPointsUp
[4+,2]

Cup BottomIsFlat,
Light,
HasConcavity,
...
ConcavityPointsUp
HandleOnTop Cup BottomIsFlat,
[0+,2] Light,
Cup BottomIsFlat, HasConcavity,
Light, ConcavityPointsUp,
HasConcavity, HandleOnSide
ConcavityPointsUp, [2+,0]
HandleOnTop
[4+,0]

16 lecture slides for textbook Machine Learning,

c T. Mitchell, McGraw Hill, 1997
 FOCL Results

Recognizing legal chess endgame positions:

 30 positive, 30 negative examples
 FOIL: 86%
 FOCL: 94% (using domain theory with 76%
accuracy)
NYNEX telephone network diagnosis
 500 training examples
 FOIL: 90%
 FOCL: 98% (using domain theory with 95%
accuracy)

17 lecture slides for textbook Machine Learning,

c T. Mitchell, McGraw Hill, 1997