Home
conchita-marotz
CSCE-642 Reinforcement Learning Chapter 5: Monte

CSCE-642 Reinforcement Learning Chapter 5: Monte

conchita-marotz

Published 2026-09-11 · 74 slides · 0 views

↓ Download

CSCE-642 Reinforcement Learning Chapter 5: Monte

1 / 1

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 1 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 2 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 3 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 4 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 5 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 6 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 7 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 8 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 9 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 10 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 11 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 12 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 13 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 14 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 15 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 16 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 17 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 18 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 19 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 20 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 21 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 22 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 23 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 24 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 25 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 26 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 27 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 28 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 29 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 30 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 31 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 32 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 33 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 34 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 35 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 36 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 37 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 38 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 39 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 40 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 41 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 42 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 43 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 44 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 45 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 46 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 47 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 48 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 49 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 50 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 51 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 52 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 53 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 54 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 55 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 56 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 57 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 58 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 59 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 60 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 61 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 62 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 63 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 64 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 65 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 66 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 67 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 68 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 69 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 70 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 71 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 72 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 73 of 74

CSCE-642 Reinforcement Learning Chapter 5: Monte - slide 74 of 74

Description

CSCE-642 Reinforcement Learning Chapter 5: Monte Carlo Methods Instructor: Guni Sharon 1 CSCE-689, Reinforcement Learning Stateless decision process Markov decision process Solving MDPs (offline) Dynamic programming Monte-Carlo Temporal

Embed code

<iframe width="560" height="315" src="https://www.docslides.com/embed/1100528" frameborder="0" allowfullscreen></iframe>

Download this presentation From Below

"CSCE-642 Reinforcement Learning Chapter 5: Monte" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.

Presentation Transcript

CSCE-642 Reinforcement Learning Chapter 5: Monte Carlo Methods Instructor: Guni Sharon 1

CSCE-689, Reinforcement Learning Stateless decision process Markov decision process Solving MDPs (offline) Dynamic programming Monte-Carlo Temporal difference Tabular methods Function approximators Policy gradient Deep RL Actor-critic 2

Reinforcement Learning 3

Offline vs. Online (RL) Offline Optimization Online Learning 4

Monte-Carlo Methods Monte Carlo methods are a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results
The underlying concept is to obtain unbiased samples from a complex/unknown distribution through a random process
They are often used in physical and mathematical problems and are most useful when it is difficult or impossible to compute a solution analytically
Weather prediction
Computational biology
Computer graphics
Finance and business
Sport game prediction 5

Value-based learning via Monte-Carlo r 6

Example: MC Evaluation Input Policy  Assume:  = 1 Observed Episodes (Training) Output Values B, east, C, -1
C, east, D, -1
D, exit, , +10 B, east, C, -1
C, east, D, -1
D, exit, , +10 E, north, C, -1
C, east, A, -1
A, exit, , -10 Episode 1 Episode 2 Episode 3 Episode 4 E, north, C, -1
C, east, D, -1
D, exit, , +10 +8 +4 +10 -10 -2 7

Problems with MC Evaluation What’s good about direct evaluation?
It’s easy to understand
It doesn’t require any knowledge of the model
It converges to the true expected values

What’s bad about it?
It wastes information about transition probabilities
Each state must be learned separately
So, it takes a long time to learn Output Values +8 +4 +10 -10 -2 If B and E both go to C with the same probability, how can their values be different? 8

Greedy MC +10 -1 +1 9

Greedy MC +10 -1 +1 10

Greedy MC +10 -1 +1 We converged on a local optimum! 11

Must explore! 12

13

MC control - example +10 -100 14

MC control - example +10 -100 15

MC control - example +10 -100 16

MC control - example +10 -100 17

MC control - example +10 -100 18

MC control - example +10 -100 19

MC control - example +10 -100 20

MC control - example +10 -100 21

MC control - example +10 -100 22

MC control - example +10 -100 23

On-policy learning +10 -1 +1 24 Estimation True value

Off-policy learning 25

Off-policy learning conditions 26

Trajectory probability 27

Importance sampling YES! 28

Importance sampling 29

Importance sampling: proof 30

Importance sampling: proof 31 Importance ratio

(ordinary) Importance sampling - example +10 -1 +1 32

Weighted importance sampling +10 -1 +1 Ordinary Importance sampling is unbiased while the weighted version is biased (initially). Ordinary Importance sampling results in high variance while the weighted version has a bounded variance Trick: normalize by the sum of importance ratios 33

Ordinary vs weighted importance sampling Estimating a black-jack state
Target policy: hit on 19 or below
Behavior policy: random (uniform)
Both approaches converge to the true value
weighted importance sampling is much better initially 34

MC control + importance sampling 35

MC control + importance sampling 36

MC control + importance sampling Going back in time 37

MC control + importance sampling Discount future rewards and add immediate reward 38

MC control + importance sampling 39

MC control + importance sampling Incremental update of Q values (waited moving average) 40

MC control + importance sampling Update target policy (greedy) 41

MC control + importance sampling 42

MC control + importance sampling 43

MC control + IS example +10 -100 44

MC control + IS example +10 -100 45

MC control + IS example +10 -100 46

MC control + IS example +10 -100 47

MC control + IS example +10 -100 48

MC control + IS example +10 -100 49

MC control + IS example +10 -100 50

MC control + IS example +10 -100 51

MC control + IS example +10 -100 52

MC control + IS example +10 -100 53

MC control + IS example +10 -100 54

MC control + IS example +10 -100 55

MC control + IS example +10 -100 56

MC control + IS example +10 -100 57

MC control + IS example +10 -100 58

MC control + IS example +10 -100 59

MC control + IS example +10 -100 60

MC control + IS example +10 -100 61

MC control + IS example +10 -100 62

MC control + IS example +10 -100 63

MC control + IS example +10 -100 64

MC control + IS example +10 -100 65

MC control + IS example +10 -100 66

MC control + IS example +10 -100 67

MC control + IS example +10 -100 68

MC control + IS example +10 -100 69

MC control + IS example +10 -100 70

MC control + IS example +10 -100 71

MC control + IS example +10 -100 72

What did we learn? 73

What next? Lecture: Temporal Difference Learning
Assignments:
Monte-Carlo Control
Monte-Carlo Control with Importance Sampling
Due by Sep-29, EOD
Quiz (on Canvas):
Monte-Carlo Control
By Sep-17, EOD
Project:
Converge on your project’s topic and scope 74

CSCE-642 Reinforcement Learning Chapter 5: Monte

Related Topics

Share

Embed code

Download this presentation From Below

Presentation Transcript