Numerical method for HJB equations. Optimal control problems and

Transcription

Numerical method for HJB equations. Optimal
control problems and differential games
(lecture 3/3)
Maurizio Falcone (La Sapienza) & Hasnaa Zidani (ENSTA)
ANOC, 23–27 April 2012
M. Falcone & H. Zidani ()
HJB approach for optimal control problems
1 / 50
Outline
1
Introduction
2
Planing Motion, reachability analysis
3
Hamilton-Jacobi approach: level set method
4
Differential games under state-constraints
2 / 50
Consider the controlled system:
ẏx (s) = f (yx (s), α(s)),
yx (0) = x,
α(s) ∈ A,
s ∈ (0, +∞),
(1)
a.e s ∈ (0, +∞).
where A is a convex compact set in Rm , (m ≥ 1).
Admissible trajectories:
S[0,τ ] (x) := {yx satisfying (1) on(0, τ ), yx (0) = x}
Under classical assumptions, the set-valued function x
S[0,τ ] (x) is
Lipschitz continuous,
∃L > 0, S[0,τ ] (x) ⊂ S[0,τ ] (z) + L|x − z|BW 1,1
∀x, z ∈ Rd .
4 / 50
Now, consider the following control problems:
ä Mayer’s problem:
V (x, t) =
inf
yx ∈S[0,t] (x)
Φ(yx (t))
ä Time minimum problem (C closed set in Rd ):
T (x) = inf t; yx (t) ∈ C, yx ∈ S[0,t] (x)
ä Supremum cost:
V ∞ (x, t) =
inf
yx ∈S[0,t] (x)
Φ(yx (t))
_
sup g(yx (θ))
θ∈[0,t]
5 / 50
Assume that g : Rd → R is Lipschitz continuous (+classical
assumptions on f )
If Φ : Rd → R is lsc (resp. Lipschitz), then V and V ∞ are lsc (resp.
Lipschitz).
When the target C is closed, the minimum time function T is lsc.
6 / 50
Mayer Problem
V (x, t) =
min
yx ∈S[0,h] (x)
V (yx (h), t − h)
h ∈ (0, t),
V (x, 0) = Φ(x)
Minimum time problem:
T (x) =
min
yx ∈S[0,h] (x)
T (yx (h)) + h
h < T (x), x 6∈ C,
T (x) = 0 x ∈ C;
Supremum cost
V ∞ (x, t) =
min
yx ∈S[0,h] (x)
V ∞ (x, 0) = Φ(x)
_
V ∞ (yx (h), t − h)
_
sup g(yx (θ))
θ∈[0,h]
g(x);
7 / 50
V (x, t) =
V (x, t) =
min
yx ∈S[0,t] (x)
Φ(yx (t))
min V (yx (h), t − h)
yx ∈S[0,h]
h ∈ (0, t).
V (x, 0) = Φ(x)
ä Suboptimality:
∀yx ∈ S[0,t] (x),
s 7−→ V (yx (s), t − s) is increasing,
ä Superoptimality
∃yx∗ ∈ S[0,t] (x),
s 7−→ V (yx∗ (s), t − s) is constant
8 / 50
Next, we derive the Hamilton-Jacobi-Bellman equation (HJB), which is
an infinitesimal version of the DPP.
ä ∂t V (x, t) + H(x, Dx V (x, t)) = 0,
V (x, 0) = Φ(x)
ä H(x, DT (x)) = 1,
T (x) = 0 on C
x ∈ Rd , t > 0;
Time-dependent HJB equation
x 6∈ C, T (x) < +∞;
Steady HJB equation
ä min(∂t V ∞ (x, t) + H(x, DV ∞ (x, t)), V ∞ (x, t) − g(x)) = 0,
x ∈ Rd , t > 0;
V ∞ (x, 0) = Φ(x) ∨ g(x)
HJB-VI ineqation
where H(x, q) := maxa∈A (−f (x, a) · q).
9 / 50
ä To each control problem is associated an adequate
Hamilton-Jacobi equation
ä A very large class of control problems can be considered within
the HJ framework (state-constrained control problem, infinite
horizon control problems, hybrid systems, impulsive control, ... )
ä The viscosity notion provides a very convenient framework for the
theoretical and numerical studies of the value function
10 / 50
ä When the (exact) value function is known, the feed-back controller can
be defined as the minimizer of the DPP.
ä This feedback can be shown to be an optimal control law
Van der Pol Problem :

ẏ1 (t) = y2




ẏ2 (t) = −y1 + y2 (1 − y12 ) + a




a(t) ∈ [−1, 1]
2
1.5
1
0.5
0
−0.5
−1
Scheme:ENO2−RK1
Target
−1.5
−2
−2
−1
0
1
2
11 / 50
ä When the (exact) value function is known, the feed-back controller
can be defined as the minimizer of the DPP.
ä This feedback can be shown to be an optimal control law
1.5
1
0.5
x2
Van der Pol Problem :


ẏ1 (t) = y2



ẏ2 (t) = −y1 + y2 (1 − y12 ) + a(t)



 a(t) ∈ [−1, 1]
0
−0.5
−1
−1.5
−1.5
−1
−0.5
0
x1
0.5
1
1.5
12 / 50
Open problem
In general, only an approximation of the value function can be
computed.
It is not clear how the feedback control behaves with respect to a small
perturbation of the value function.
13 / 50
Numerical methods
Semi-Lagrangian methods: based on the DPP (Falcone, Ferretti,
Jakobsen, Grüne, Kushner-Dupuis, ...)
PROS: no CFL condition for stability (⇒ adaptative schemes)
CONS: non-local
Finite difference methods: approximation of the gradient by FD
(Crandall/Lions, Barles, Souganidis)
CONS: needs CFL condition for stability
PROS: local method =⇒ can be parallelized
PROS: non-monotone variants are proposed to get numerically
"high-order"
• -monotone schemes (R. Abgrall)
• ENO, WENO (Osher, Shu, ... )
• Discontinuous Galerkin, direct DG (Cockburn, Shu, Cheng&Shu,
Bokanowski’12)
• Anti-diffusive schemes, Ultra-Bee (Megdich-Bokanowski-HZ’10,
Bokanowski-Cristiani-HZ’10)
14 / 50
Recent developments and ongoing works:
• "Curse of dimentionality free methods" : max-plus algebra
(McKeneaney, Akian, Gaubert, Sridharan, ...)
• Sparse grids method (Bokanowski/Klompmaker/Garke/Griebel 12’)
15 / 50
ENO2-RK1 scheme (and example of numerical Hnum )
void HJB_FD::ENO2_RK1(double t, double deltat, double* vin, double* vout){
int i,j,d;
double vi,v1,v2,v3,v4,vv,h;
for(j=0;j<mesh->inn_nbPoints;j++){
i
= rank[j];
vi
= vin[i];
for(d=0;d<dim;d++){
v1 = vin[i mesh->out_neighbors[d]];
v3 = vin[i - 2*mesh->out_neighbors[d]];
v2 = vin[i +
mesh->out_neighbors[d]];
v4 = vin[i + 2*mesh->out_neighbors[d]];
h = mesh->Dx[d];
vv = (v2-2.*vi+v1)*divdx[d]*divdx[d];
Dvnum[2*d] = (vi-v1)*divdx[d] + h*.5*minmod((vi-2.*v1+v3)*divdx[d]*divdx[d],vv);
Dvnum[2*d+1]= (v2-vi)*divdx[d] - h*.5*minmod((vi-2.*v2+v4)*divdx[d]*divdx[d],vv);
}
vout[i] = vi - deltat * (*this.*Hnum)((mesh->*(mesh->getcoords))(i),Dvnum,t);
}
}
inline double Hnum(const double* x, const double* v, double t)
{
double z=0., p[DIM], amax[DIM];
int i;
for(i=0;i<DIM;i++){
amax[i]=1.; //- a maximal bound of the dynamics
p[i]=(v[2*i] + v[2*i+1])/2.;
z += amax[i]*(v[2*i+1] - v[2*i])/2.;
}
return H(x,p)-z; //- H(x,Nabla u) is the Hamiltonian
}
16 / 50
HJ parallel Library (Binope)
by O. Bokanowski, A. Désilles, J. Zhao, H. Zidani
http://www.ensta-paristech.fr/~zidani/BiNoPe_HJ/presentation.html
Finite Differences solver (ENO, UltraBee)
C++, parallel (MPI/OpenMP)
works in any dimension (limited to machine’s capacity)
Semi-Lagrangian solver
C++ (OpenMP)
works in any dimension (limited to machine’s capacity)
17 / 50
Reachable (or Attainable) set
ä The reachable set Rf (x; t) from x at time t is the set of all points
of the form yx (τ ), where yx ∈ S[0,t] (x):
Rf (x; t) := {yx (τ ) | yx ∈ S[0,t] (x), τ ∈ [0, t]}.
ä The reachable set from X is defined by:
RfX (t) := ∪x∈X Rf (x; t).
19 / 50
Let C be a closed target set (in our examples, C is safe)
Capture Basin (or Backward reachable set)
ä The Capture Basin CaptCt , at time t, is the set of all initial positions
x from which a trajectory yx ∈ S[0,t] (x) can reach the target C.
20 / 50
ä Does there exist a trajectory leading from a state in initial set
X to a state in the target C, during some finite time horizon?
ä Once an obstacle has been detected by suitable sensors
(e.g. radar, pursuer), can a collision be avoided?
ä Sometimes we have no control over input signal (noise,
actions of other agents, unknown system parameters, ...): it
is safest to consider the worst-case.
21 / 50
Different approaches for computing the reachable sets
ä set-valued integration schemes [Saint-Pierre’91, Baier’95], optimal
control techniques [Varaiya’00, Baier et al.’07]
ä external and inner ellipsoidal techniques [Kurzhanski and
Varaiya’00,’01,’02]
ä discretization methods for nonlinear problems with state
constraints [Chahma’03, Beyn an Rieger’07]
ä Optimal controller design: level set method [Osher-Sethian’91,
Falcone-et-al.’05, Mitchell’07, Bokanowski-HZ’07,
Bokanowski-Forcadel-HZ’10]
22 / 50
Optimization-based controller design
Minimum-time control problem:
T (x) = inf t; yx (t) ∈ C, yx ∈ S[0,t] (x) .
The sublevels of the minimum function T correspond to the
Capture Basins of the target C:
CaptCt = {x ∈ Rd | T (x) = t}.
When the minimum time function T is continuous, it can be
characterized as the unique viscosity solution of the HJB
equation:
H(x, DT (x)) = 1,
T (x) = 0 in C,
x 6∈ C, T (x) < +∞;
where H(x, q) := maxa∈A (−f (x, a).q).
23 / 50
Unfortunately ...
ä The continuity of T is equivalent to small controllability of the
system around the target: let ηx be the normal to C,
min f (x, α) · ηx < 0,
α∈A
∀x ∈ ∂C.
ä This (restrictive) controllability property is not satisfied in
several examples (e.g. Zermelo problem).
ä In the general case, the minimum time function is only lsc. In
this case, the approximation of the minimum value function
becomes more difficult.
24 / 50
Zermelo problem
25 / 50
Optimal control problem. Level set approach
Define the signed distance function Φ(x) = dC (x),
and consider the following control problem:
V (x, t) =
inf
yx ∈S[0,t] (x)
Φ(yx (t))
27 / 50
ä V is Lipschitz continuous and satisfies (Crandall-Lions’84):
∂t V (x, t) + H(x, DV (x, t)) = 0,
V (x, 0) = Φ(x),
where H(x, q) := supa∈A (−f (x, a) · q)
ä For every t ≥ 0, CaptCt = {x ∈ Rd ; V (x, t) ≤ 0};
ä The minimum time function T : Rd → R+ ∪ {+∞} is lsc.
Moreover, we have:
T (x) = inf{t ≥ 0; x ∈ CaptCt } = min{t ≥ 0; V (x, t) ≤ 0}.
ã Φ can be any function satisfying
Φ(x) ≤ 0 ⇐⇒ x ∈ C.
28 / 50
ä The level set approach can be used even when
the minimum time function is discontinuous!
The value function V is Lipschitz continuous!
ä The level set approach can be extended to more
general situations: differential games, avoidance
of obstacles, moving target and/or obstacles, ...
29 / 50
General setting: differential games under state
constraints
Let α ∈ A be a controlled input, and β ∈ B an uncontrolled input
(perturbance). Consider the trajectory:
ẏx (s) = f (yx (s), α(s), β(s)), s ∈ (0, 1),
yx (0) = x,
Let (Kθ )θ≥0 be a family of closed set (of constraints). Consider a
game involving two players.
I
I
The first player wants to steer the system from the initial position at
point x to the target C and by staying in K (and using his/her input
α(t) ∈ A)
while the second player tries to steer the system away from C or
from K (with his/her input β(t) ∈ B).
Assume θ 7−→ Kθ is usc.
31 / 50
Non anticipative strategies
We define the set of non-anticipative strategies for the first player, as
follows:
Γ := a : B → A, ∀(β, β̃) ∈ B and ∀s ∈ [0, ∞),
β(θ) = β̃(θ) a.e. θ ∈ [0, s] ⇒
a[β](θ) = a[β̃](θ) a.e. θ ∈ [0, s]
.
32 / 50
ä For τ ≥ 0,
a[β],β
CaptK (τ ) := {x ∈ Rd | ∃α, ∀β, yx
a[β],β
(s) ∈ Ks , and yx
(τ ) ∈ C}
ä Again, let Φ(x) = dC (x) and consider the control problem:
ϑ(x, τ ) := min
max
Φ(y (τ )).
a[β] β|y a[β],β (t)∈K
t
x
Then
CaptK (τ ) = {x ∈ Rd , ϑ(x, τ ) ≤ 0}
T (x) = min{τ ≥ 0, ϑ(x, τ ) ≤ 0}.
ä For controlled systems lacking controllability assumptions, the
characterization of ϑ by means of HJB equations is not an easy
task !!! (Ref: Soner’86, Ishii-Koike’91, Frankowska’91,
Altarovici-Bokanowski-HZ’12)
33 / 50
An other formulation: exact penalisation
ä Let g(x, θ) = dKθ (x) for any θ ≥ 0 and x ∈ Rd . Then, consider :
ϑg (x, τ ) := inf max Φ(yx (τ ))
a[.]∈Γ
β
_
a[β],β
max g(yx
(θ), θ).
θ∈[0,τ ]
ä ϑg is the unique continuous viscosity solution
min ∂ϑg (x, τ ) + H(x, Dx ϑg (x, τ )), ϑg (x, τ ) − g(x, τ )) = 0,
ϑg (x, 0) = max(Φ(x), g(x, 0)).
where H(x, q) := supa∈A minb∈B (−f (x, a, b) · q)
ä And
Captt = {x ∈ Rd , ϑg (x, τ ) ≤ 0},
T (x) = min{τ ≥ 0, ϑg (x, τ ) ≤ 0}.
34 / 50
Example1: One player, fixed obstacles
35 / 50
Example2: Zermelo problem with obstacles
x 0 = Vboat cos(θ) + Vcurrent − ay 2
y 0 = Vboat sin(θ)
36 / 50
Zermelo problem with obstacles: feedback control law
2
1.5
x 0 = Vboat cos(θ) + Vcurrent − ay 2
y 0 = Vboat sin(θ)
1
0.5
0
−0.5
−1
Scheme:ENO2−RK1
Obstacle
Target
−1.5
−2
−3
−2.5
−2
−1.5
−1
−0.5
0
0.5
1
1.5
2
37 / 50
Exemple: Ariane V
Objectif
Minimize the ergol consumption to
steer the (given) payload MCU to
the GTO (or GEO).
Collaboration with Cnes (projet OPALE 2007-2010)
38 / 50
−−→
The physical model involves 7 state variables, the position OG of
→
−
the rocket in the 3D space, its velocity v and its mass m.
eK
er
G
→
−
v
γ
`
O
eJ
e`
G
L
eI
χ
eL
→
−
Projection of v on the frame (er , eL , e` )
→
−
−
→
The forces acting on the rocket are: Gravity P , Drag FD , Thrust
−
→
→
−
FT , and Coriolis Ω .
Newton Law:
→
−
→
− −
→ −
→
→
− →
→
−
→
− −−→
dv
−
m
= P + FD + FT − 2m Ω ∧ v − m Ω ∧ ( Ω ∧ OG),
dt
39 / 50
PHASE C (HJB)
m1
m2
m3
GEO
GTO
PHASE B
(transport)
PHASE A(HJB)
m1
m3 m2
40 / 50
The related equation
State variables:
r =altitude
v =modulus of the velocity
γ=angle between the direction earth-rocket and the direction of the
rocket’s velocity.
L= latitude
`= longitude
χ= azimuth
m= masse of the engine
Control:
α=angle between the thrust direction and the direction of the rocket’s
velocity.
41 / 50
ṙ = v cos γ
FD (r , v ) FT (r , v , a)
+
cos α
m
m
Ω2 r cos `(cos γ cos ` − sin γ sin ` sin χ)
g(r ) v
FT (r , v , a)
γ̇ = sin γ
−
−
sin α
v
r
vm
r
−2Ω cos ` cos χ − Ω2 v cos `(sin γ cos ` − cos γ sin ` sin χ)
v̇ = −g(r ) cos γ −
v sin γ cos χ
r
cos `
v
`˙ = sin γ sin χ
r
v
χ̇ = − sin γ tan ` cos χ − 2Ω(sin ` − cotanγ cos ` sin χ)+
r
r sin ` cos ` cos χ
Ω2
v
sin γ
L̇ =
42 / 50
â The plane of motion is the equatorial plane ` ≡ 0, and χ ≡ 0.
ṙ = v cos γ
FD (r , v ) FT (r , v , a)
+
cos α + Ω2 r cos γ
v̇ = −g(r ) cos γ −
m
m
r
g(r ) v
FT (r , v , a)
γ̇ = sin γ
−
−
sin α − 2Ω − Ω2 sin γ
v
r
vm
v
v
L̇ = sin γ
r
ṁ = −b(m(t))
43 / 50
The rocket’s mass
ä The evolution of the mass can be summarized as follows
Phase 0 & 1
ṁ1 (t) = −βEAP
ṁ2 (t) = −βE1
ṁ3 (t) = 0
Phase 2
ṁ1 (t) = 0
ṁ2 (t) = −βE1
ṁ3 (t) = 0
Phase 3
ṁ1 (t) = 0
ṁ2 (t) = 0
ṁ3 (t) = −βE2
where βEAP , βE1 and βE2 are the mass flow rates for the boosters,
the first and the second stage.
ä At the changes of phases, we have a (not negligible) discontinuity
in the rocket’s mass.
44 / 50
The control problem can be formulated as (for a fixed payload)
Minimize tf
(r , v , γ, m, α) satisfy the state equation
α(t) ∈ [0, π/2]
a.e. t ∈ (0, tf ),
(r (tf ), v (tf ), γ(tf )) ∈ C,
Q(r (t), v (t))α(t)) ≤ Cs
for t ∈ (0, tf ),
m(tf ) = Mp .
where the target C corresponds to the GTO orbit, and the function Q is
the dynamic pressure.
45 / 50
ã The Capture Basin is wide
+ We introduce "physical" state constraints to define the
computational domain
ã Due to the CFL condition, the time step is very small
+ Adaptative time discretization
ã "Different scales" for the state variables:
r = r0 (ex − 1) + rT
+ Change of variable:
v = v0 (ey − 1) + vT
46 / 50
200
0
gamma (rad)
speed (m/s)
400
0
10000
8000
6000
4000
2000
500
1000
time (sec)
1.5
mass (ton)
altitude (km)
GTO target
1
0.5
0
500
1000
time (sec)
0
500
1000
time (sec)
600
400
200
0
0
500
1000
time (sec)
Figure: Full trajectory using the HJB minimal time value function
Reference trajectory, final mass:
HJB trajectory, final mass (after reconstruction):
mT = 21.57 (t)
mT = 22.50 (t)
47 / 50
200
0
gamma (rad)
speed (m/s)
400
0
10000
8000
6000
4000
2000
500
1000
time (sec)
1.5
mass (ton)
altitude (km)
GTO target
1
0.5
0
500
1000
time (sec)
0
500
1000
time (sec)
600
400
200
0
0
500
1000
time (sec)
Figure: Full trajectory using the HJB minimal time value function
Reference trajectory, final mass:
HJB trajectory, final mass (after reconstruction):
mT = 21.57 (t)
mT = 22.50 (t)
48 / 50
"Collision analysis for an UAV".
E. Crück, A. Desilles, HZ.
AIAA Guidance, Navigation, and Control, 2012
"A general Hamilton-Jacobi framework for nonlinear state-constrained control problems".
A. Altarovici, O. Bokanowski and HZ
ESAIM:COCV, 2012
"Minimal time problems with moving targets and obstacles". O. Bokanowski and HZ.
18th IFAC World Congress, Milan, 2011
"Deterministic state constrained optimal control problems without controllability assumptions". O. Bokanowski, N.
Forcadel and HZ
ESAIM: COCV, 17(04), pp. 975–994, 2011
"An efficient data structure and accurate scheme to solve front propagation problems". O. Bokanowski, E. Cristiani and
HZ
J. of Scientific Computing, 42(2), pp. 251–273, 2010
"Reachability and minimal times for state constrained nonlinear problems without any controllability assumption". O.
Bokanowski, N. Forcadel and HZ
SIAM J. Control and Optimization, vol. 48(7), pp. 4292-4316, 2010
"Convergence of a non-monotone scheme for Hamilton-Jacobi-Bellman equations with discontinuous initial data".
O. Bokanowski, N. Megdich and HZ.
Numerische Mathematik, 115(1), pp. 1–44, 2010
"An anti-diffusive scheme for viability problems"
O. Bokanowski, S. Martin, R. Munos and HZ.
Applied Num. Methematics, 56(9), pp. 1147–1162, 2006
49 / 50
... many thanks for your attention!
50 / 50

Numerical method for HJB equations. Optimal control problems and

Transcription

Similar documents

Oxford - Tidewater Vacations

Notice of 10th Extra-Ordinary General Meeting

Match-ID