A NUMERICAL METHOD FOR SOLVING LINEAR–QUADRATIC CONTROL PROBLEMS WITH CONSTRAINTS1

The paper is devoted to the optimal control problem for a linear system with integrally constrained control function. We study the problem of minimization of a linear terminal cost with terminal constraints given by a set of linear inequalities. For the solution of this problem we propose two-stage numerical algorithm, which is based on construction of the reachable set of the system. At the first stage we find a solution to finitedimensional optimization problem with a linear objective function and linear and quadratic constraints. At the second stage we solve a standard linear-quadratic control problem, which admits a simple and effective solution.


Introduction
The optimal control problems under integrally constrained controls were studied in many papers (see, for example, [1, 5, 7-9, 12, 13]). In [2,3] the authors considered the linear control system with integrally constrained control. They studied the problem of minimization of a linear terminal cost under linear terminal constraints and proposed a saddle-point method to solve it.
We propose here a two-stage numerical algorithm for the solution of above optimal control problem. It is based on constructing the reachable set of the control system, and we use here the well-known result that this set for a linear control system with integral quadratic constraints on controls is an ellipsoid in the state space. Then, at the first stage, we find a minimum of a linear function on the intersection of the polyhedron and ellipsoid. This problem may be solved numerically in different ways. At the second stage, we solve the standard linear-quadratic control problem with fixed endpoints of the trajectory, this problem has a simple and effective solution in the linear case. The typical unpleasant feature of the optimal control problem with terminal cost is a nonuniqueness of solutions, which always takes place if the endpoint of the optimal trajectory belongs to the interior of the reachable set. This leads to additional difficulties in the construction and implementation of numerical algorithms. The method proposed here avoids these problems.

Notation and problem statement
We use the following notations. By A ⊤ we denote the transpose of a real matrix A, 0 stands for a zero vector of appropriate dimension, I is an identity matrix. For x, y ∈ R n let (x, y) = x ⊤ y be the inner product, x ⊤ = (x 1 , . . . , x n ), x = (x, x) 1 2 be the Euclidean norm, and B r (x): B r (x) = {x ∈ R n : x −x ≤ r} be a ball of radius r > 0 centered atx. For a set S ⊂ R n let ∂S, intS, clS, coS be a boundary, an interior, a closure, and a convex hull of S respectively; ∇g(x) is the gradient of a function g(x) at the point x, ∂f ∂x (x) is the Jacobi matrix of a vector-valued function f (x). For a real k × m matrix A a matrix norm is denoted as A k×m . The symbols L 1 , L 2 and C stand for the spaces of summable, square summable and continuous functions respectively. The norms in these spaces are denoted as · L 1 , · L 2 , · C .
We consider a linear control system with integral constraints on a controlṡ Let the control constraints are defined by the integral quadratic inequality where µ > 0 is a given number. For any u(·) ∈ L 2 there exists a unique absolutely continuous solution x(t) of system (1.1), which is defined on interval [t 0 , t 1 ]. Assume that m × n-matrix D, vectors d ∈ R m and c ∈ R n are given. We consider here the following optimal control problem for system (1.1): under constraints u(·) ∈ U, Dx(t 1 ) ≤ d.
(1.4) Definition 1. The reachable set G(t 1 ) of system (1.1) at time instant t 1 is a set of all states x(t 1 ) that can be reached by the trajectories of (1.1) corresponding to controls u(·) ∈ U : The considered optimization problem may be split into two following subproblems.
Problem 1 (the first stage): to find x * ∈ R n that solves the finite-dimensional optimization problem Here the reachable set G(t 1 ) is an ellipsoid in R n those parameters are calculated effectively. Thus, the optimization Problem 1 may be solved by employing the methods of linear or convex programming.
Problem 2 (the second stage): to find a control u(·) ∈ U that steers the trajectory x(t) of (1.1) to point x(t 1 ) = x * and minimizes functional J(u(·)).

Description of reachable sets
A solution of (1.1) at time instant t 1 has the form Take an arbitrary vector l ∈ R n , l = 0 and calculate the maximum of inner product (l, Here is an inner product of functions v(·), u(·) in the Hilbert space L 2 , and symmetric matrix W (t) is defined by the equality Differentiating the last equality in t we get the following matrix differential equation for W : It is known (see, for example, [10]), that system (1.1) is completely controllable on [t 0 , t 1 ], if and only if W (t 1 ) is positive definite. In this case G(t 1 ) is a nondegenerate ellipsoid If the system is not completely controllable, the reachable set is a degenerate ellipsoid (ellipsoid, lying in a subspace of dimension less than n). It is obvious, that x ∈ ∂G(t 1 ) if and only if there exists a vector l = 0 such that (l, x) = max y∈G(t 1 ) (l, y), and hence, a control u(·) steering the system trajectory to point x satisfies the relation where v(t) is defined above. If the system is completely controllable, v(·) = 0 and the equality (2.3) uniquely determines is a nontrivial solution of the adjoint differential equationṗ(t) = −A ⊤ (t)p(t) and the relations hold. Define a Hamiltonian H(p, t, x, u) by the expression Equating to zero the gradient of H(p(t), t, x(t), u) in u, from the concavity of the Hamiltonian in u we get that control u(t) = B ⊤ (t)p(t) satisfies the maximum principle where p(t) is a nontrivial solution of the adjoint differential equatioṅ This form of the maximum principle corresponds to the problem of minimization of a convex functional J(u(·)) on solutions of linear system (1.1) Since for a linear-convex optimal control problem the maximum principle provides necessary and sufficient optimality conditions, a control u(t) = B ⊤ (t)p(t), found from the maximum principle, solves problem (2.4). Inversely, let u(·) be the solution to problem (2.4) and let J(u(·)) = µ 2 . Then we will have x ∈ G(t 1 ). Indeed, if we assume that x / ∈ ∂G(t 1 ), then Hence, the control system can be transferred to the point x by the control v(·), for this control we have J(v(·)) ≤ ν 2 < µ 2 which contradicts to the optimality of u(t). Thus, we come to the following statement.
Assertion 1. Let system (1.1)4 be completely controllable. In order to control u(·) steers a trajectory of system (1.1) to point x, lying on the boundary of the reachable set G(t 1 ), it is necessary and sufficient that this control solves the extremal problem (2.4) and the minimum of functional J equals to µ 2 .

The algorithm of solving the optimal control problem
Having the parameters of the reachable set given, let us describe the algorithm of the solution of the problem. Consider system (1.1) and assume that it is completely controllable. Then, at the first stage, we should solve the Problem 1 The problem (3.1) is a linear programming problem with an additional constraint, which is defined by an inequality with a positive definite quadratic form. This problem may be solved by various algorithms. For example, in [11] the authors proposed a finite convergent algorithm for solution of the problem of type (3.1).
The second way of finding the solution concerns with description of the ellipsoidal reachable set via its support function: where the support function has the form ψ(l) = µ l ⊤ W (t 1 )l + (x, l), with S = {l : l = 1}. Then, the problem (3.1) may be written as follows (3. 2) The problem (3.2) is a semi-infinite linear programming problem [6], which can be solved by a number of effective numerical algorithms. Choosing a finite grid of N vectors l i ∈ S, we can approximate the problem (3.2) by the following linear programming problem with a finite number of constraints Let a solution x * of problem (3.1) ((3.2), (3.3)) be obtained (this solution is unique as a rule).
Then we come to the next stage: to find the solution of the next problem For completely controllable system this solution does exist and is unique. It may be obtained from the maximum principle: Represent p(t) as follows then we have Thus, to find an optimal control it is sufficient: 1) to find a vector p 1 from the linear equation 2) to integrate adjoint equation (3.5) with boundary condition p(t 1 ) = p 1 and substitute p(t) into the formula for u(t). We can put from the very beginning x(t 1 ) =x + W (t 1 )p 1 and solve a semi-infinite linear programming problem in the dual variables p 1 The last form is more convenient, it may be used even in the case of degenerate matrix W (t 1 ). Really, it is known [8] that if there exists a control from L 2 that steers the control system from x 0 to x * , than control u(·) solving the problem (3.4) is a linear combination of columns of matrix for some p 1 ∈ R n . Substituting the control u(t) into equality (3.6) we get (3.7). Thus, the following theorem holds.  Consider an illustrative example of optimal control problem for a linear control system described by the equationṡ x 3 = u with t 0 = 0, t 1 = 2π and x 0 = (1, 0, 0) ⊤ . We consider the integral quadratic constraints on controls given by the inequality (1.2) with µ 2 = 1.
Assume a matrix D and a vector d that determine the terminal constraints are The graph of the optimal control is shown in Figure 1 by the black line. Here x * is the interior point of the reachable set, so there are infinitely many admissible control inputs that steer the trajectories to point x * . Among these inputs the considered control input has a minimal value of the integral functional J(u(·)) which equals to 0.3185. Consider another case and put c = (1, 0, 0) ⊤ . In this case we have the following solution Here (x * −x) ⊤ W −1 (2π)(x * −x) = 1, hence x * belongs to the boundary of the reachable set. In this case there exists a unique optimal control input which is shown in Figure 1 by the blue line.

Further generalizations
Consider here the generalized statement for the previous problem. We assume that integral constraints restrict simultaneously a control and a trajectory of system (1.1) as follows J(u(·)) = 1 2 where Q(t) is a nonnegative definite and R(t) is a positive definite matrix for every t ∈ [t 0 , t 1 ]. Suppose Q(t) and R(t) to be the measurable and bounded matrix functions.
Assume that a pair (A(t), B(t)) is completely controllable on [t 0 , t 1 ]. Let x(t 0 ) = x 0 be fixed and let x ∈ G(t 1 ). The set of trajectories satisfying inequality (5.1) is a compact set in C. Hence, there exists the solution to the problem this solution is unique due to strict convexity of the functional J(u(·)).

Consider a Hamiltonian
According to the maximum principle (see, for example, [4,10]) there exist (p 0 , p(·)) = 0 such that u(t) maximizes a Hamiltonian, hence we have ∂H ∂u = 0 anḋ Assuming p 0 = 0, we get p(·) = 0. Then p(t) is a nonzero solution of the equatioṅ and the condition ∂H/∂u = 0 implies the equality p ⊤ (t)B ≡ 0, this contradict to the controllability conditions. Thus, p 0 = 0, so we can take p 0 = 1. In this case we have Substituting control (5.2) into equations of the control system we geṫ Thus, (5.3) is a linear homogeneous system of differential equations The solution of (5.4) with the initial state x(t 0 ) = x 0 , p(t 0 ) = p 0 has the form where Y (t) is the fundamental matrix of system (5.4) satisfying the initial condition Y (t 0 ) = I.

Conclusion
We consider an optimal control problem for a linear system with integrally constrained control, with a linear terminal cost and with terminal constraints given by a set of linear inequalities. This problem is, in fact, the ill-posed problem because of nonuniqueness of optimal control, which always takes place if the end point of the optimal trajectory belongs to the interior of the reachable set of the control system. We propose here a simple numerical algorithm for solving the optimal control problem, which uses a known explicit description of the reachable sets for linear systems with integral quadratic constraints on control functions. The algorithm is based on the reduction of considered problem to the solution of a finite-dimensional convex programming problem in primal or dual variables. This method allows to avoid difficulties related to nonuniqueness of optimal control.