Optimization of Zero-Order Markov Processes with Final Sequence of States

In this paper the zero-order Markov processes with final sequence of states X and unit transition time are analyzed. The evolution time T (p) of these systems is studied, where p represents the distribution of the states of the system. The problem of minimization the expectation E(T (p)) is considered. This problem is reduced to a geometric program, which is efficiently solved using convex optimization based on interior-point methods. The main idea of the proof is to show that the expression E(T (p))+1 is a posynomial function in variables which represent the components of distribution of the states that participate in final sequence of states. For some particular cases the explicit solution is obtained.


Introduction
The discrete Markov processes are often used as mathematical models that describe various applied actual problems from many important domains: economy, technique, biology, industry, medicine and others. Based on these stochastic systems, the researchers obtain new numerical methods and algorithms for solving the complex problems of the society.
The stochastic systems with final sequence of states generalize the discrete Markov processes. For these systems the stopping condition is defined, i.e., the system stops when it passes through given final sequence of states. Unlike Markov processes, the evolution time of stochastic systems with final sequence of states is finite and depends on realization of stopping condition. For these systems the problem of determining the main probabilistic characteristics of evolution time of the system is interesting. This problem was studied in [14] and [15], where polynomial algorithms based on the main properties of homogeneous linear recurrences, generating function and numerical derivation of regular rational fractions were obtained. The main generalizations of these systems were presented in [12], [13] and [15].
In this paper the zero-order Markov processes with final sequence of states and unit transition time are studied. These systems represent a particular case of stochastic systems with final sequence of states. They are also called stochastic systems with final sequence of states and independent states or strong memoryless stochastic systems with final sequence of states, since at every discrete moment of time the state of the system does not depend on previous states.
For these stochastic processes the efficient method for minimizing the expectation of the evolution time is elaborated. This method is based on geometric programming approach, that reduces the problem to the case of convex optimization using interior-point methods.
Geometric programming was introduced in 1967 by Duffin, Peterson, and Zener in [6]. Wilde andBeightler in 1967 andZener in 1971 contributed with many results (see [24] and [25]) referred of many extensions and sensitivity analysis. A short history of geometric programming was presented by Peterson in [19].
A geometric program represents a type of optimization problem described by objective and constraint functions that have a special form. This form is characterized by following rules: the objective function and left-hand side of inequality constraints need to be posynomials and left-hand side of equality constraints needs to be monomials. Also, in standard form, the geometric program is a minimization problem with right-hand side of all constraints equal to 1 and all inequality constraints containing the "≤" operator. A good tutorial on geometric programming was presented in [3].
First numerical methods based on solving a sequence of linear programs were elaborated by Avriel et al. [2], Duffin [5] and Rajpogal and Bricker [21]. Nesterov and Nemirovsky in 1994 described the first interior-point method for geometric programs and demonstrated the polynomial time complexity in [18]. Recent numerical approaches were presented by Andersen and Ye [1], Boyd and Vandenberghe [4], Kortanek [10] and Ungureanu [23]. In this paper a discrete stochastic system L with the set of possible states V , where |V | = N ≤ ∞, is considered. The state of the system at every discrete moment of time t = 0, 1, 2, . . . is denoted by v(t) ∈ V . The transition time of the system from the state u to another state v at every moment of time t is equal to 1.
On the set V a distribution function p : V → [0, 1] is defined, where p(v) represents the probability with which the system L starts its evolution from the state v ∈ V . Also, the transition of the system from arbitrary state u ∈ V to another state v ∈ V at every moment of time t is performed with probability p(v) ∈ [0, 1].
The finishing evolution of the system is conditioned by passing consecutively through fixed sequence of states X = (x 1 , x 2 , . . . , x m ) ∈ V m . Let T be the stopping time of the discrete stochastic system L. The value T represents the evolution time of the stochastic system L.
The system L represents a zero-order Markov process with final sequence of states X and distribution of the states p. For this system the moments of positive integer order n of the evolution time are studied in [14] and [15]. In particular case the expectation of evolution time is obtained.
Next, we consider that the distribution p is not fixed. So, we have the zero-order Markov process L(p) with final sequence of states X and distribution of the states p, for every p from the set The problem is to determine the optimal distribution p * ∈ P, that minimizes the expectation of the evolution time T (p) of the stochastic system L(p), i.e.
In the following chapters we will show how this problem can be solved using geometric programming.

Geometric programming
The definition of geometric program requires the definition of monomials and posynomials. Posynomials are closed under addition, multiplication, and nonnegative scaling. Posynomials are not the same as polynomials, since: • a polynomial's exponents must be non-negative integers, but a posynomial's exponents can be arbitrary real numbers; • a polynomial's coefficients can be arbitrary real numbers, but a posynomial's coefficients must be positive real numbers.
Using these notions, we can define a geometric program in the following way.

Definition 3.
A geometric program is an optimization problem of the form: In order to efficiently solve a geometric program we need to convert it to a convex optimization problem. Efficient solution methods for general convex optimization problems were developed by Boyd and Vandenberghe in [4]. The conversion is based on a logarithmic change of variables y k = ln x k , k = 1, s and a logarithmic transformation of the objective and constraint functions. The obtained convex optimization problem has the form: Unlike the original problem, the obtained problem looks more complicated, but it is convex and can be solved very efficiently using standard interior-point methods (see [3] and [4]). Also, the interior-point methods for solving geometric programs are very robust, require no starting point or other parameters and always find the globally optimal solution of the problem.

Homogeneous linear recurrences
Next, we remind some definitions and notations from [12], [14] and [15] regarding homogeneous linear recurrences. We consider a subfield K of the field C. ∈ K m such that q m−1 ̸ = 0 and a n = m−1 ∑ k=0 q k a n−1−k , for all n ≥ m.

Definition 5. The vector q from previous definition represents the generating vector and the vector I
We introduced the following notations:

200
Optimization of Zero-Order Markov Processes with Final Sequence of States is the set of non-degenerated homogeneous linear m-recurrent sequences on the set K;

• G[K][m](a) represents the set of generating vectors of length m of the sequence a ∈ Rol[K][m]
.
a n z n is called generating function of the sequence a = (a n ) ∞ n=0 ⊆ C and the function G a n z n is called partial generating function of order t of the sequence a.

Definition 7. Let a ∈ Rol[K][m] and q ∈ G[K][m](a).
For the sequence a we will consider the unitary characte- Also, the following notation is introduced: The next theorem presents the formula for the generating function.

Distribution of evolution time
The zero-order Markov processes with final sequence of states were described in Section 2 and the evolution time T (p) of the system was defined. Let us consider the distribution a = (P(T (p) = n)) ∞ n=0 . In [14] and [15], the properties of sequence a were studied. We obtained that a ∈ Rol[R][m] for p(x 1 ) ̸ = 1 and p(x j ) ̸ = 0, j = 1, m. Also, we obtained formulas for the initial state I

Simplified version of Algorithm 1
Next, we eliminate the parameters from Algorithm 1. The following theorem holds.

Theorem 2. If one assumes the notations from Section 4.1, then the formula
Proof. The particular case k ≤ t(s) − 2 is trivial from Algorithm 1. Next, we consider the case k ≥ t(s). We have: In this way we obtained the assertion.
Using Theorem 2, Algorithm 1 can be rewritten in the following form.

The values
3. For each s = 1, m the following steps are executed: are calculated.

Expectation of evolution time
The following theorem holds.

Theorem 3. The expectation of the evolution time T (p)
of zero-order Markov process L(p) can be determined using the formula Proof. From Theorem 1 and Algorithm 2, we obtain the formula for the generating function: Since from Algorithm 2 we have q 0 = 1 − π 1 v m,0 and q k = −π 1 v m,k , k = 1, m − 1, we obtain The proof is complete.

General case of the problem
In this subsection we present the main results. We prove that the problem of optimization the expectation of evolution time can be reduced to a geometric program.  and for all 1 ≤ s < m. The last relation implies x t(s+1)+1 , . . . , x s ) = (x 1 , x 2 , . . . , x s+1−t(s+1) ), for all 1 ≤ s < m. Since the value t = t(s) represents the least positive integer number that verifies the relation we have t(s) ≤ t(s + 1), for all 1 ≤ s < m. So, the sequence (t(s)) m s=1 represents a monotonically increasing sequence.
Then, we obtain which represents the assertion of the lemma.
Proof. This Lemma is obtained from Theorem 2 and Lemma 4. Applying Theorem 2, we obtain that there exists the index l such that Next, using Lemma 4, we have Also, from Theorem 2, it is easy to see that the difference between indexes is not changed at every step of recurrence, i.e., the relation m * − k * = m − k holds. The condition 0 ≤ k * < t(m * ) represents the stopping rule of the recurrent formula from Theorem 2.
Proof. From Theorem 2 and Theorem 3 we have Let be z mk = w m w m * z m * k * , where the indexes m * and k * were defined in Lemma 5.
The following scenarios are possible: In this case we have In this case we have k * +1 = t(m * )−1, that implies z m * k * = −w t(m * )−1 and z m * ,k * +1 = w t(m * )−1 . We obtain It is easy to see that this expression represents a posynomial in the variables π 1 , π 2 , . . . , π m . In this case we have k * +1 = t(m * )−1, that implies z m * k * = −w t(m * )−1 and Let be r ≥ 2. Using Lemma 3, we obtain: Next, applying Lemma 2, Lemma 4 and Lemma 5, we have the relation that implies the formula In this way we obtain that the expression mw m + g mk represents a posynomial in the variables π 1 , π 2 , . . . , π m .
In this case we have • The case k * = t(m * ) − 1.
Theorem 4 is very important. It shows how the problem described in Section 2 can be reduced to the geometric program If π * = (π * (x)) x∈Y represents the optimal solution of this geometric program, then p * = (p * (x)) x∈V represents the optimal solution of the initial problem, where In obtained geometric program we actually have ∑ x∈Y p(x) = 1. Indeed: • The expression ∑ x∈Y p(x) represents a posynomial, but is not a monomial in variables p(x), x ∈ Y .
We have to mention that the relation p * (x) = 0, ∀x ∈ V \Y , is true from the statement of the initial problem. Indeed, if we have p * (x * ) > 0 for at least one state x * ∈ V \Y , then the states that are components of final sequence of states have a lower probability of realization and, in consequence, the evolution time is greater and the solution is not optimal.

Optimal solutions for particular cases
In this subsection some particular cases of the problem are presented. The explicit optimal solutions are obtained. The following theorems hold.
Proof. Let be t(m) = 2. We have . . = x m and the stochastic system L(p) represents a stochastic system with final critical state x 1 . We obtain that So, the optimal solution is p * = (p * (x)) x∈V , where p * (x 1 ) = 1 and p * (y) = 0, for all y ∈ V \{x 1 }. It is easy to observe that this fact implies the relation E(T (p * )) = m − 1.

Theorem 6.
If t(m) = m + 1, then the components p * (y), y ∈ V , of the optimal solution p * are direct proportionally with the multiplicities m(y), y ∈ V , of the respective states in final sequence of states X and the minimal value of the expectation of evolution time is Proof. Let be t(m) = m + 1. We have From Theorem 3, the formula for the expectation of evolution time is E(T (p)) = −1 + w −1 m . So, we have the optimization problem which is equivalent with the optimization problem For each state x ∈ Y we denote by m(x) its multiplicity in final sequence of states X. We obtain the optimization problem Next, we apply the classical optimization method involving the partial derivatives of the objective function. We chose a state y * ∈ Y and we denote by Y * the set Y \{y * }. We have p( For each y ∈ Y * , we obtain where h(y, y * ) = (p(y)) m(y) (p(y * )) m(y * ) . We have ∂h(y, y * ) ∂p(y) = m(y)(p(y)) m(y)−1 (p(y * )) m(y * ) − m(y * )(p(y)) m(y) (p(y * )) m(y * )−1 is equivalent with the system m(y)p(y * ) − m(y * )p(y) = 0, ∀y ∈ Y * , which implies the relation We obtain the formula from which we have p(y * ) = m(y * ) m . Next, we obtain the components of the optimal solution of the problem: For E(T (p * )), we have the formula , which represents the assertion of the theorem. 6 Practical implementation 6

.1 Expectation of the evolution time
For implementing the established method, we used the Wolfram Mathematica package.
Initially we need to set the input parameters of the method: the sequence V ar of variables that represents the distribution of the states from final sequence of states, the sequence Xind of indexes (in sequence V ar) of the variables from final sequence of states and, optionally, the approximation eps of the zero value in numerical calculus for selecting real solutions when is applied the classical optimization method. For example, if the final sequence of states of the stochastic system is X = (x 1 , x 1 , x 2 , x 2 , x 1 , x 1 ) and the approximation of the zero value is ε = 10 −4 , then we need to run the following instructions in Wolfram Mathematica :

Optimal distribution of the evolution time
Having the formula obtained in previous subsection, we can apply the geometric programming approach or other methods for optimizing the evolution time of zeroorder Markov process.
Next, we present two alternative versions using Wolfram Mathematica package. These versions are easier to implement than geometric programming, but they are less effective, since Wolfram Mathematica package uses general methods for solving nonlinear optimal problems. Additionally, we mention that the second version works without warnings and errors only when the system of nonlinear equations containing partial derivatives of the expectation of evolution time has a finite non-empty set of admissible solutions.
First implementation is based on integrated Minimize function. For the example analyzed in previous subsection, we need to run the following instructions: The second implementation is based on the classical optimization method. First we need to express an arbitrary variable by others from the relation ∑ p∈V ar p = 1.
For the studied example, we have the instruction: Next, we need to run the following instructions:

A two-dimensional numerical example
For the example considered in Section 6.1, by running the programs from Section 6.1 and 6.2, we obtain E(T (p)) = −1 + p −2 1 + p −1 1 + p −4 1 p −2 2 , the optimal distribution p * = (0.673694, 0.326306) and the optimal evolution time E(T (p * )) = 48.2808. The results obtained by using both versions from Section 6.2 are the same.
The graph of the function E(T (p)) of the variable p2 is obtained by running the following instruction:
The graph of the function E(T (p)) of the variables p2 and p3 is obtained by running the following instruction:

Conclusions
In this paper the following results were established: • analysis of the zero-order Markov processes with final sequence of states and unit transition time; • optimization of the existing algorithm for determining the distribution of evolution time; • optimization of the expectation of evolution time by applying geometric programming approach; • implementation of the obtained methods by using Wolfram Mathematica package; • verification of the obtained scientific results by testing the implemented versions for various numerical examples.
These results can be used in the theory of Markov processes and its applications.