Learning What is learning? - PowerPoint Presentation

Learning What is learning? What are the types of learning?Why aren’t robots using neural networks all the time? They are like the brain, right? Where does learning go in our operational architecture? 1 Zazzle.com

2 Learning Objectives Describe the 4 major types of learning by example and the associated techniquesDescribe the following learning techniques: induction, support vector machines, Q-learning, artificial neural networks Describe simulated annealing and genetic algorithms and how they relate to learning but are not learning algorithms per se Define new term problem, overfitting

Outline 3 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning ArchitectureSim Ann./GAObservations Types Unsupervised Artificial neural networks Supervised Induction Support Vector Machines Reinforcement Q-Learning Architecture: where it goes Simulated Annealing and Genetic Algorithms Observations

Learning (Russell & Norvig) An agent improves its performance on future tasks after making observations about the world In order to know which AI technique to use, you need to know:Which component is to be improvedWhat prior knowledge the agent already hasWhat representation is used for the data and the componentWhat feedback is available to learn from 4 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

Types of Learning By example/experience (common for robots)Unsupervised learning Supervised learningReinforcement learningSemi-supervised learningBy reasoning (very hard, not common)Case based/explanation based learningBy AnalogyNot learning but often associated with itSimulated annealing, genetic algorithms 5 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

Unsupervised learning 6 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning ArchitectureSim Ann./GAObservations

Unsupervised Learning The robot learns patterns in the input even though no explicit feedback is supplied Major techniquesClustering/pattern recognitionNeural networks 7 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement-Q-learningArchitecture Sim Ann./GA Observations

Clustering http://www.youtube.com/watch?v=rhallml-juk What features to track?It wasn’t always right 8 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement-Q-learningArchitectureSim Ann./GA Observations

9 Artificial Neural Networks Also used when you have large set of measurements and no model AND and XOR connectionsEx. If input a>= 0.3 AND b>= 0.6 then fireAllowing hidden layers supports more complexity inputs outputs hidden layers Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

inputs outputs hidden layer a w w w g w g w w g a a a w w w g w w w g a w g a i i i i i a a 10

inputs outputs hidden layer o 3 g g g g i 1 i 2 i 4 i 5 i 3 w 1 o 1 g o 2 g h 1 h 2 h 3 w 2 w 3 w 4 w 5 w 6 w 7 w 8 w 9 w 10 w 11 w 12 w 13 w 14 11

Artificial Neural Networks http://www.youtube.com/watch?v=99DOwLcbKl8 12 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learningArchitectureSim Ann./GAObservations

Artificial Neural Networks http://www.youtube.com/watch?v=99DOwLcbKl8 It took thousands of trialsEven though unsupervised, there was an explicit performance metric (moving forward)Have to put limits on the number of hidden layers, number of outputs (parametrized vs. non-parametrized models) 13 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement-Q-learning Architecture Sim Ann./GA Observations

Unsupervised Learning The robot learns patterns in the input even though no explicit feedback is supplied Major techniquesClustering/pattern recognitionNeural networksThings robots learn (so far)Terrain classificationProbabilistic models of reliability of sensors for different environmentsMovements, action selection 14 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

Supervised learning 15 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning ArchitectureSim Ann./GAObservations

Supervised Learning the agent observes some example input–output pairs and learns a function, h, that maps from input to outputThe output is labeled as positive or negativeMajor techniquesInductionSupport vector machines (SVM)Decision trees 16 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

Induction: inferring a concept from labeled input-output pairs 17 O O O O O O O O Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

18 A hypothesis h Concept 1 Concept 2 O O O O O O O O Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

Concept 1 Concept 2 O O O O O O O O 19

20 h is consistent if fits all data O O O O O O O O Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

O O O O O O O O O O O O O O O O 21

22 Overfitting: consistent but doesn’ t hold for unobserved events O O O O O O O O Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

O O O O O O O O 23

Support Vector Machines Rather than try to come up with the function h that best separates concepts from the training data (empirical loss), come up with an h that best separates the statistical distribution of data for the concepts (generalization loss)Transform the space into a higher dimension which makes it easier to find h 24 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement-Q-learningArchitecture Sim Ann./GA Observations

SVM Example 25 O O O O O O O O Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

26 O O O O O O O O

27 Decision Trees & ID3 If you have lots of training data, you can use statistics to determine the best choice or action Ex. Given a set of measurements (observations) about the terrain and weather, what is the right mobility control?But it would be even better if you learned which set of measurements were valuable and the order in which to get the measurementsA decision tree Weather? Rain Dry Terminate Terrain Type? Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

weather time of day road difficulty FAILS FAILSFAILSSUCCEEDSFAILSSUCCEEDSroad lanes SUCCEEDS FAILS foggy rainy clear day twilight night curves straight curves and hills two lanes 4 or more lanes 28

Supervised Learning the agent observes some example input–output pairs and learns a function, h, that maps from input to outputThe output is labeled as positive or negativeMajor techniquesInductionSupport vector machines (SVM)Decision treesThings robots learn (so far)NavigationObject recognitionCommands 29 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

Reinforcement Learning 30 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning ArchitectureSim Ann./GAObservations

Reinforcement Learning the agent observes some example input–output pairs and learns a function, h, that maps from input to outputThe output is labeled as positive or negativeMajor techniquesUtility functionQ learning 31 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

Utility Function Methods First Learning program: Samuel’s Checker Value (utility) of a board positionx1 number of black pieces on the boardX2 number of red pieces on the boardX3 number of black kings on the boardX4 number of red kings on the boardX5 number of black pieces threatenedX6 number of red pieces threatenedV^(b)= w0+w1x1+w2x2+w3x3+w4x4+w5x5+w6x6 Learn the weights by win/lose games 32 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

What if You Can’t Estimate Utility? http://www.youtube.com/watch?v=m2weFARriE8 33 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learningArchitectureSim Ann./GAObservations

What if You Can’t Estimate Utility? http://www.youtube.com/watch?v=m2weFARriE8 Problem of how to explore space effectively/quicklyLots of trials requiredMistakes seem obviousWhich is why you want to take advantage of a priori knowledge! 34 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement-Q-learningArchitecture Sim Ann./GA Observations

Specifics of Q-Learning http://www.youtube.com/watch?v=4vV4SNEdLt8&feature=related 35 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learningArchitectureSim Ann./GAObservations

Specifics of Q-Learning http://www.youtube.com/watch?v=4vV4SNEdLt8&feature=related Large number of trialsHow to set the weights, discount 36 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement-Q-learningArchitectureSim Ann./GAObservations

Apprenticeship Learning: Let Human Explore the Space http://www.youtube.com/watch?v=M-QUkgk3HyE 37 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement-Q-learningArchitectureSim Ann./GAObservations

Apprenticeship Learning: Let Human Explore the Space http://www.youtube.com/watch?v=M-QUkgk3HyELet 2 hours of “observing” humans fly serve as the exploring the state spaceApply Q-learningNote: human didn’t use an optimal policy! 38 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement-Q-learning Architecture Sim Ann./GA Observations

Reinforcement Learning the agent observes some example input–output pairs and learns a function, h, that maps from input to outputThe output is labeled as positive or negativeMajor techniquesUtility functionQ learningThings robots learn (so far)Most popular robotic learning techniqueHow to fly, navigate, control 39 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

Where does learning go? 40 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning ArchitectureSim Ann./GAObservations

41 Where Does Learning Go? ENVIRONMENT Percept Response SENSE ACT ACTUATORS SENSORS WORLD MODEL State, a priori Symbol-ground PLAN

42 Mitchell’s Learning Agent ENVIRONMENT Percept Response SENSE ACT ACTUATORS SENSORS WORLD MODEL State, a priori Symbol-ground Performance Element PLAN Learning Element Problem Generator Critic LEARN Performance Element : computes the next action Problem Generator: provides the training examples Learning Element: produces the new weights for choosing actions Critic : produces the rating or score of that action

43 But Doesn’t Fit Robots That Well ENVIRONMENT Percept Response SENSE ACT ACTUATORS SENSORS WORLD MODEL State, a priori Symbol-ground Performance Element PLAN Learning Element Problem Generator Critic LEARN Online learning techniques are nascent Doesn’t capture learning skills which get “pushed” to the reactive layer

ENVIRONMENT Percept Response SENSE ACT ACTUATORS SENSORS WORLD MODEL State, a priori Symbol-ground Performance Element PLAN Learning Element Problem Generator Critic LEARN 44

Simulated Annealing and Genetic Algorithms 45 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement-Q-learningArchitectureSim Ann./GAObservations

46 Searching for an Answer State space Objective Function G Test cases Simulated annealing Jump lots at the beginning (like molecules in hot metals) Jump less as time goes on (temperature cools) Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

47 Genetic Algorithms Duplicate evolutionary search; h(n) =fitness functionEx. 8-queens Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement-Q-learningArchitecture Sim Ann./GA Observations

3 2 7 5 2 4 1 1 48 8 Queens Notation Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learningArchitecture Sim Ann./GA Observations

49 Genetic Algorithms Position of each queen by column “ Score ” for P(reproduction) Crossover points randomly selected Mix ‘ em up! New state Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

50 GA Crossover Example Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

Observations and wrapup 51 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning ArchitectureSim Ann./GAObservations

Observations Learning by example is just one type of learning but the one AI has been most successful with Training is hardGet the right, tractable number of examples is hardNote: Can’t validate with the same examples you tested withLearning “new terms” is extremely difficultOnline learning is rare; the norm is offline learning or learning in simulationPeople learn quickly, often in one step which is another area of machine learningLearning can be applied to any aspect of a robot, but it doesn’t fit in one “location” 52 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations

Returning to Questions What is learning?Improving performance on future tasks after making observations about the world What are the types of learning?There are many types of learning but learning by example/experience is most common in robotsLearning by example has 4 major types: unsupervised, supervised, reinforcement, and semi-supervisedWhy aren’t robots using neural networks all the time? They are like the brain, right?Artificial neural networks are a simplification of neural nets in the brainThey require thousands to millions of examples plus some way of specifying number of hidden layersWhere does learning go in our operational architecture?Since there is no one component that can be learned, there is no one place in the architecture for “learning”, instead it has instances through out 53 Types Unsupervised - ANN Supervised -Induction -SVM Reinforcement -Q-learning Architecture Sim Ann./GA Observations