EE Winter  Lecture  Linear Quadratic Stochastic Control linearquadratic stochastic control problem solution via dynamic programming   Linear stochastic system linear dynamical system over nite time h
247K - views

EE Winter Lecture Linear Quadratic Stochastic Control linearquadratic stochastic control problem solution via dynamic programming Linear stochastic system linear dynamical system over nite time h

N is the process noise or disturbance at time are IID with 0 is independent of with 0 Linear Quadratic Stochastic Control 52 brPage 3br Control policies statefeedback control 0 N called the control policy at time roughly speaking we choo

Download Pdf

EE Winter Lecture Linear Quadratic Stochastic Control linearquadratic stochastic control problem solution via dynamic programming Linear stochastic system linear dynamical system over nite time h




Download Pdf - The PPT/PDF document "EE Winter Lecture Linear Quadratic Sto..." is the property of its rightful owner. Permission is granted to download and print the materials on this web site for personal, non-commercial use only, and to display it on your personal computer provided you do not modify the materials and that you retain all copyright notices contained in the materials. By downloading content from our website, you accept the terms of this agreement.



Presentation on theme: "EE Winter Lecture Linear Quadratic Stochastic Control linearquadratic stochastic control problem solution via dynamic programming Linear stochastic system linear dynamical system over nite time h"— Presentation transcript:


Page 1
EE363 Winter 2008-09 Lecture 5 Linear Quadratic Stochastic Control linear-quadratic stochastic control problem solution via dynamic programming 5–1
Page 2
Linear stochastic system linear dynamical system, over finite time horizon: +1 Ax Bu , t = 0 , . . . , N is the process noise or disturbance at time are IID with = 0 is independent of , with = 0 Linear Quadratic Stochastic Control 5–2
Page 3
Control policies state-feedback control: = 0 , . . . , N called the control policy at time roughly speaking: we choose input after knowing the current state,

but before knowing the disturbance closed-loop system is +1 Ax B ) + , t = 0 , . . . , N , . . . , x , u , . . . , u are random Linear Quadratic Stochastic Control 5–3
Page 4
Stochastic control problem objective: =0 Qx Ru with R > depends (in complex way) on control policies , . . . , linear-quadratic stochastic control problem : choose control policies , . . . , to minimize (‘linear’ refers to the state dynamics; ‘quadratic’ to the objective an infinite dimensional problem: variables are functions , . . . , Linear Quadratic Stochastic Control 5–4
Page 5
Solution via

dynamic programming let be optimal value of objective, from on, starting at ) = min ,..., Qx Ru subject to +1 Ax Bu we have ) = (expectation over Linear Quadratic Stochastic Control 5–5
Page 6
can be found by backward recursion: for , . . . , ) = Qz + min Rv +1 Az Bv expectation is over we do not know where we will land, when we take optimal policies have form ) = argmin Rv +1 Ax Bv Linear Quadratic Stochastic Control 5–6
Page 7
Explicit form let’s show (via recursion) value functions are quadratic, with f orm ) = , t = 0 , . . . , N, with = 0 now assume that +1 ) = +1 +1

Linear Quadratic Stochastic Control 5–7
Page 8
Bellman recursion is ) = Qz + min Rv (( Az Bv +1 Az Bv ) + +1 Qz Tr WP +1 ) + +1 min Rv + ( Az Bv +1 Az Bv we use +1 ) = Tr WP +1 same recursion as deterministic LQR, with added constant optimal policy is linear state feedback: ) = +1 +1 (same form as in deterministic LQR) Linear Quadratic Stochastic Control 5–8
Page 9
plugging in optimal gives ) = , with +1 +1 +1 +1 +1 Tr WP +1 first recursion same as for deterministic LQR second term is just a running sum we conclude that are same as in deterministic LQR strangely,

optimal policy is same as LQR, and independent of Linear Quadratic Stochastic Control 5–9
Page 10
optimal cost is Tr XP ) + Tr XP ) + =1 Tr WP interpretation: is optimal cost of deterministic LQR, with = 0 Tr XP is average optimal LQR cost, with = 0 Tr WP is average optimal LQR cost, for = 0 = 0 Linear Quadratic Stochastic Control 5–10
Page 11
Infinite horizon choose policies to minimize average stage cost = lim =0 Qx Ru optimal average stage cost is Tr WP ss where ss satisfies the ARE ss ss ss ss ss optimal average stage cost doesn’t depend on Linear Quadratic

Stochastic Control 5–11
Page 12
(an) optimal policy is constant linear state feedback ss where ss ss ss ss is steady-state LQR feedback gain doesn’t depend on Linear Quadratic Stochastic Control 5–12
Page 13
Example system with = 5 states, = 2 inputs, horizon = 30 chosen randomly; scaled so max = 1 = 10 ∼N (0 , X = 10 ∼N (0 , W = 0 Linear Quadratic Stochastic Control 5–13
Page 14
Sample trajectories sample trace of and 10 15 20 25 30 −4 −2 10 15 20 25 30 −1 blue: optimal stochastic control, red: no control ( = 0 Linear Quadratic

Stochastic Control 5–14
Page 15
Cost histogram cost histogram for 1000 simulations 100 200 300 400 500 600 700 100 200 100 200 300 400 500 600 700 100 200 100 200 300 400 500 600 700 100 200 100 200 300 400 500 600 700 100 200 pre ol nc Linear Quadratic Stochastic Control 5–15
Page 16
Comparisons we compared optimal stochastic control ( = 224 ) with ‘prescient’ control decide input sequence with full knowledge of future disturbanc es , . . . , u computed assuming all are known pre = 137 ‘open-loop’ control , . . . , u depend only on , . . . , u computed assuming = 0 ol =

423 no control = 0 nc = 442 Linear Quadratic Stochastic Control 5–16