Approximation Methods Hongning Wang CS@THU Outline

celsa-spraggs

Published 2026-09-28 · 36 slides · 0 views

↓ Download

Approximation Methods Hongning Wang CS@THU Outline

Approximation Methods Hongning Wang CS@THU Outline - slide 1 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 2 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 3 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 4 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 5 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 6 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 7 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 8 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 9 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 10 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 11 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 12 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 13 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 14 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 15 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 16 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 17 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 18 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 19 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 20 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 21 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 22 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 23 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 24 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 25 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 26 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 27 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 28 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 29 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 30 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 31 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 32 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 33 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 34 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 35 of 36

Approximation Methods Hongning Wang CS@THU Outline - slide 36 of 36

Description

Approximation Methods Hongning Wang CSTHU Outline Value function approximation On-policy prediction with approximation On-policy control with approximation Off-policy learning with approximation CSTHU RL2025-Fall 2 Tabular methods for RL

Download this presentation From Below

"Approximation Methods Hongning Wang CS@THU Outline" is the property of its rightful owner. Permission is granted to download and print the materials on this website for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.

Presentation Transcript

Approximation Methods Hongning Wang
CS@THU

Outline Value function approximation
On-policy prediction with approximation
On-policy control with approximation
Off-policy learning with approximation CS@THU RL2025-Fall 2

Tabular methods for RL Enumerate and store all state(-action) values in a big table
Recall MP2 CS@THU RL2025-Fall 3

Tabular methods for RL CS@THU RL2025-Fall 4

Function approximation for RL CS@THU RL2025-Fall Figure credit: David Silver, “Value Function Approximation” 5

Supervised machine learning Classification
Discrete response Regression
Continuous response CS@THU RL2025-Fall 6

Supervised machine learning Fundamental assumptions
Independent and identically distributed instances
Empirical risk minimization
Typically under a stationary distribution CS@THU RL2025-Fall Both are violated in an RL setting! 7

Value function approximation CS@THU RL2025-Fall 8

Value function approximation CS@THU RL2025-Fall Goal of value function learning? Guide policy optimization Mean square error is NOT necessarily the best objective for value function approximation, whose aim is to help policy optimization. 9

Incremental update Update the approximation after every action taking
Any online optimization method would work
The objective function is not necessarily convex
Stochastic gradient descent is typically the choice CS@THU RL2025-Fall 10

Incremental update CS@THU RL2025-Fall 11

Incremental update CS@THU RL2025-Fall 12

Linear models One of the most well-studied approximation methods

Guaranteed convergence under gradient descent
Needs an unbiased gradient estimation, e.g., MC methods CS@THU RL2025-Fall 13

Linear models under semi-gradients Under TD(0) semi-gradient CS@THU RL2025-Fall When the estimation converges 14

Recap: Bellman equation for value function CS@THU RL2025-Fall 15

Recap: value function approximation CS@THU RL2025-Fall 16

Linear models under semi-gradients Under TD semi-gradient CS@THU RL2025-Fall Approximation error obtained by TD Approximation error obtained by MC 17

Incremental update CS@THU RL2025-Fall 18

Many other different types of approximations Non-linear models
Regression trees
Neural networks, a.k.a, Deep RL
Kernel methods CS@THU RL2025-Fall Almost all methods we have learned in machine learning apply here! All aforementioned methods apply to action value function approximation! 19

Batch update CS@THU RL2025-Fall Solution: from online update to batch update 20

Experience replay CS@THU RL2025-Fall Figure credit: MC.AI Prioritize by TD error (proportionally) 21

Prioritized experience replay CS@THU RL2025-Fall Figure credit: Schaul et al., “Prioritized experience replay”, ICLR’2016 Q-learning with tables Q-learning with linear approximation 22

Control with value approximation Under generalized policy iteration CS@THU RL2025-Fall 23

On-policy control with approximation CS@THU RL2025-Fall 24

Sarsa with action value approximation CS@THU RL2025-Fall Three possible actions:
full throttle forward (+1)
full throttle reverse (-1)
zero throttle (0)
Reward is -1 on all time steps until reaching the goal 25

Sarsa with action value approximation CS@THU RL2025-Fall 26

Off-policy control with approximated function value Now we have shifted target value and target distribution CS@THU RL2025-Fall 27

Target value correction Importance sampling
Importance weight

Gradient correction CS@THU RL2025-Fall The previous learnt off-policy methods, e.g., Q-learning, Treeback also apply for this purpose, as they directly provide us the target value. 28

Challenges in off-policy control with approximation Behavior policy is unaware of the quality of approximation
A one-dimensional case CS@THU RL2025-Fall 29

Baird’s counterexample CS@THU RL2025-Fall Reward is zero everywhere 30

Baird’s counterexample CS@THU RL2025-Fall Deadly triad:
Function approximation
Bootstrapping
Off-policy learning 31

Linear value-function geometry CS@THU RL2025-Fall What is a better objective for off-policy learning? Expectation of TD error! 32

Takeaways Function approximation generalizes value estimation across states
On-policy prediction and control are typically performed as SGD for parameter estimation
Off-policy with value approximation is much more challenging CS@THU RL2025-Fall 33

Suggested readings Chapter 9: On-policy Prediction with Approximation
Chapter 10: On-policy Control with Approximation
Chapter 11: Off-policy Methods with Approximation CS@THU RL2025-Fall 34

Emphatic-TD methods CS@THU RL2025-Fall 35

Emphatic-TD methods CS@THU RL2025-Fall Baird’s counterexample 36

Approximation Methods Hongning Wang CS@THU Outline

Related Topics

Share

Embed code

Download this presentation From Below

Presentation Transcript