Ockham's razor

The most likely hypothesis is the simplest one that is consistent with all observations.
Propounded by the 13th century philosopher William of Ockham.
Training set
Used in supervised learning a training set is a set of problem instances !described as a set
of properties and their values" together with a classification of the instance.
Inductive learning
# form of supervised learning. $iven a set of e%amples !inputs and their evaluations via
some function" the system tries to appro%imate the evaluation function.
Q-learning
&earning an action-value function that is learning the e%pected utility of taking a
particular action in a given state.
memoization
#ccumulating a database of input'output values to avoid having to reason again about an
already(solved problem.
Types of Learning
There are si% main types of learning.
Speed-up learning
# type of deductive learning that re)uires no additional input but improves the agent*s
performance over time. There are two kinds rote learning and generalization !e.g. +,&". -ata
caching is an e%ample of how it would be used.
Learning by taking advice
-eductive learning in which the system can reason about new information added to its
knowledge base. .c/arthy proposed the 0advice taker0 which was such a system and
T+12+31#3 4-avis 15678 was the first such system.
Learning from examples
1nductive learning in which concepts are learned from sets of labeled instances.
Clustering
Unsupervised inductive learning in which 0natural classes0 are found for data instances
as well as ways of classifying them. +%amples include /O,W+, #UTO/&#33.
Learning by analogy
1nductive learning in which a system transfers knowledge from one database into a that of
a different domain.
Inductive Learning
1nductive learning is a kind of learning in which given a set of e%amples an agent tries to
estimate or create an evaluation function. .ost inductive learning is supervised learning in
which e%amples provided with classifications. !The alternative is clustering." .ore formally an
example is a pair (x, f(x)) where x is the input and f(x) is the output of the function applied to x.
The task of pure inductive inference !or induction" is given a set of e%amples of f to find a
hypothesis h that appro%imates f.
9 f!x" is the target function
9 #n example is a pair 4x f!x"8
9 &earning task: find a hypothesis h such that h!x" f!x" given a training set of e%amples
- ; <4x
i
f!x
i
" 8= i ; 1>?N
9 Construct h so that it agrees with f.
9 The hypothesis h is consistent if it agrees with f on all observations.
9 Ockhams razor: Select the simplest consistent hypothesis.
In supervise learning! the learning element is given the correct value of the function for
particular inputs! an changes its representation of the function to try to match the information
provie by the feeback. "ore formally! we say an e#ample is a pair $#!f$#%%! where # is the
input an&$it% is the output of the function applie to #. The task of pure inuctive inference $or
inuction% is this: given a collection of e#amples! return a function h that appro#imates. The
function h is calle a hypothesis.
"easuring the performance of the learning algorithmCollect a large set of
e#amples.'ivie it into two is(oint sets: the training set an the test set.)se the learning
algorithm with the training set as e#amples to generate a hypothesis *."easure the percentage
of e#amples in the test set that are correctly classifie by *.+epeat steps , to - for ifferent
sizes of training sets an ifferent ranomly selecte training sets of each size.
( ) ( ) C A C B A
{ } ( ) C A n y n
N n
=
= ... 1
" ! " ! x D