04
what is a concept? 4 A Concept could be defined as a Boolean-valued function C defined over the larger set
Example: a function defined over all animals whose value is true for birds and false for every other animal.
c(bird) = true
c(cat) = false
c(car) = false
c(pigeon) = true<br>
05
What is concept learning? 5 Given a set of examples labeled as members (positive) or non-members (negative) of a concept:
Infer the general definition of this concept
Approximate c from training examples:
Infer the best concept-description from the set of all possible hypotheses<br>
06
Example of a Concept Learning task Concept: Good days on which my friend enjoys water sport
(values: Yes, No)
Task: predict the value of ”Enjoy Sport” for an arbitrary day
based on the values of the other attributes
Result: classifier for days 6<br>
07
representing hypotheses Many possible hypothesis (h) representations
The simplest h could be represented as a conjunction of constraints over the instance attributes
Each constraint can be:
a specific value (e.g., Water = Warm)
?: don’t care/any value (e.g., Water=?)
no value allowed (e.g., Water=Ø)
Example: hypothesis h
Sky Temp Humid Wind Water Forecast
< Sunny ? ? Strong ? Same >
We say h(x)=1 for a day x, if x satisfies the description 7<br>
08
most general/specific hypothesis every day is a positive example
No day is a positive example 8<br>
09
Prototypical concept learning task GIVEN:
Instances X: Possible days, each described by the attributes
sky, AirTemp, Humidity, Wind, Water, Forecast
Target function c: EnjoySport: X{0,1}
Hypotheses H: Conjunctions of constraints on attributes
<?, Cold, High, ?, ?, ?>
Training examples D: positive and negative examples of the target function
<x1,c(x1)>,<x2,c(x2)>,…,<xn, c(xn)>
DETERMINE:
A hypothesis h in H with h(x)=c(x) for all x in D 9<br>
10
the inductive learning hypothesis Any hypothesis found to approximate the target function well over the training examples, will also approximate the target function well over the unobserved examples.
Tom Mitchell
Assumptions for Inductive Learning Algorithms:
The training sample represents the population
The input attributes permit discrimination 10<br>
11
Number of Instances, concepts, hypotheses 11 Learning can be viewed as a task of searching a large space
Sky: Sunny, Cloudy, Rainy
AirTemp: Warm, Cold
Humidity: Normal, High
Wind: Strong, Weak
Water: Warm, Cold
Forecast: Same, Change
#distinct instances : 3*2*2*2*2*2 = 96
#distinct concepts : 296
#syntactically distinct hypotheses : 5*4*4*4*4*4=5120 (general and specific cases are added)
#semantically distinct hypotheses : 1+4*3*3*3*3*3=973<br>
12
Concept learning as Search Learning can be viewed as a task of searching a large space
Different learning algorithms search this space in different ways! 12<br>
13
General-to-Specific Ordering of Hypothesis Many algorithms rely on ordering of hypothesis
Consider two hypotheses:
h1=(Sunny, ?, ?, Strong, ?,?)
h2= (Sunny, ?,?,?,?,?)
h2 imposes fewer constraints than h1: it classifies more instances x as positive h(x)=1
h2 is more general than h1!
How to formalize this? 13<br>
14
General-to-Specific Ordering of Hypothesis Many algorithms rely on ordering of hypothesis
Consider two hypotheses:
h1=(Sunny, ?, ?, Strong, ?,?)
h2= (Sunny, ?,?,?,?,?)
h2 imposes fewer constraints than h1: it classifies more instances x as positive h(x)=1
h2 is more general than h1!
How to formalize this?
h2 is more general than h1 iff h2(x)=1 h1(x)=1
We note it h2 ≥g h1 14<br>
15
Instance, Hypotheses, and More-General 15 The ≥g relation is important as it provides a structure over the hypothesis space. x1=< Sunny,Warm,High,Strong,Cool,Same> x2=< Sunny,Warm,High,Light,Warm,Same> h2=< Sunny,?,?,?,?,?> h3=< Sunny,?,?,?,Cool,?> h1=< Sunny,?,?,Strong,?,?><br>
16
16 FIND-S Algorithm Initialize h to the most specific hypothesis in H:
For each positive training instance x:
For each attribute constraint ai in h:
If the constraint is not satisfied by x
Then replace ai by the next more general
constraint satisfied by x
Output hypothesis h<br>
17
FIND-S h = < , , , , , >
h = <Sunny, Warm, Normal, Strong, Warm, Same>
h = <Sunny, Warm, ? , Strong, Warm, Same>
h = <Sunny, Warm, ? , Strong, ? , ? > 17<br>
18
FIND-S h = <Sunny, Warm, ? , Strong, ? , ? > Prediction 18<br>