A Modified Perry's Conjugate Gradient Method Based on Powell's Equation for Solving Large-Scale Unconstrained Optimization

It is known that the conjugate gradient method is still a popular method for many researchers who are focused in solving the large-scale unconstrained optimization problems and nonlinear equations because the method avoids the computation and storage of some matrices so the memory’s requirements of the method are very small. In this work, a modified of Perry conjugate gradient method which fulfills a global convergence with standard assumptions is shown and analyzed. The idea of new method is based on Perry method by using the equation which is founded via Powell in 1978. The weak Wolfe–Powell search conditions is used to choose the optimal line search, under the line search and suitable conditions we prove both descent and sufficient descent conditions. In particular, numerical results show that the new conjugate gradient method is more effective and competitive when compared to other of standard conjugate gradient methods including: CGHestenes and Stiefel (H/S) method, CG-Perry method CGDai and Yuan (D/Y)method. The comparison is completed under a group of standard test problems with various dimensions from the CUTEst test library and the comparative performances of the methods are evaluated by total the number of iterations and the total number of function evaluations.


Introduction
Conjugate gradient (CG) methods have been used to solve large-scale unconstrained optimization problem which is presented as: Where ℎ( ) is smooth and twice continuously differentiable function over , is a number of variables for our problem. In general Conjugate gradient (CG) methods are generate a sequence of estimates to the minimum of ℎ( ), by iterative formula: - However, > 0 is a steplength and must be achieved the following Wolfe-Powell search conditions: - And using a search direction: where is a scalar determines the conjugate gradient methods (CG-methods), and we denote∇ℎ( ) which is gradient of ℎ( ) by .
At current, the most effective formulas of are the following: In which, the definition of / is due to Hestenes and Stiefe (H/S, 1952) [6], / is due to Fletcher and Reeves (F/R, 1964) [4,5], PRP is due to Polak-Ribiere-Polyak (PRP, 1969) [7], is suggested by Perry(1978) [8] and in (1999) / is established by Dai-Yuan [2]. For iterative mathematics method the global convergence is one of requirement for it , and it mean by global convergence is any sequence generated via the iterative methods will either achieved after finite steps or consist of a subsequence which is converged to stationary point of the problem ( ℎ( ) ) from a given initial point ( 0 ) , lim →∞ (‖∇ℎ( + )‖) = 0 ) [11]. For Conjugate gradient the global convergence is investigated by researchers , Zoutendijk G. in 1970 proved that the Fletcher and Reeves method is global convergence in case that the line search is exact [13]. In Polak-Ribire method global convergence is described by Powell [9] when ℎ ( ) is strongly convex and the line search is also exact, but after that Powell demonstrated that the Polak-Ribire strategy with exact line search could circle infinitely without convergent to a required point. The same result applies to the Hestenes-Stiefel method.
Many efforts have been made in few recent years to design new formulas for conjugate gradient method which are not only satisfied global convergence but also improve numerical performance for method. Remainder the conjugate gradient methods have many application in real life In our work, we found a new formula for CG-method which is satisfied the global convergence in section 3.3 and the new formula with weak Wolfe-Powell generate a descent direction at each iteration in section 3.2.

Derivation of the New Conjugacy Coefficient
In this section, we present a new formula for conjugate gradient method as a result improve the parameter of in (6) by using the equation which is proposed by Powell in (1978) [10].
The idea of our new formula is based on replacing the equation in Perry formula (3) with the equation * * , and getting Hence, we get the new parameter conjugacy: - we observe that , if the orthogonal condition is satisfied i.e.

+1
= 0 then the parameter in (14) become as a following: (16) Based on Powell's Equation for Solving Large-Scale Unconstrained Optimization So, we obtain the modified Perry Conjugacy Coefficient, and get new direction:-

The Algorithm of New Conjugate Gradient Method:
The New CG-method is summarized below by following steps: Step 0: Let = 0, select 0 ∈ is an initial point, ∈ ℤ and > 0 .

Descent and Sufficient Descent Conditions
To ensure that the new algorithm is convergence we needed to prove the following theorems: -Theorem3.2.1:-Consider the sequences of { , , } are generated by New CG-method then the search direction { }satisfies the descent property Assume the induction hypothesis that for a particular ( )meaning < 0 is true for any > 0 , now we prove (18) is hold at + 1, from the definition of new direction we have Multiply both sides of (20) by +1 from right to get, Since, is parameter greater than zero so: it is clear that the first two terms of (24) refer to Hestenes and Stiefel method generates the descent directions, now we need to prove that the third term of (24) is less than or equal to zero. Noticeably , all of , ( +1 ) 2 , ‖ ‖ , and ‖ +1 ‖are positive and ∈ (0,1) . Noteworthy that ,.
In case, when the orthogonal property is satisfy, we see Multiply both sides (25) by +1 , we get The terms −‖ +1 ‖ 2 + ‖ +1 ‖ 2 +1 proved is the descent property by Dai and Yuan [2] and we proved term < 0) in above . So the descent property is satisfied when +1 = 0 Hence, we obtain +1 +1 ≤ 0 (27) the proof is completed by induction.
Theorem3.2.2: -let the sequences of { } and { } are generated by our proposed method of modified CG satisfies the descent property for all new CG-method is also hold the sufficient condition:

Proof :
Observably, the Hestenes and Stiefel method generates descent directions so, the term (24) is less than or equal to zero, consequently we can rewrite the (24) as following:- Multiply and divide the right side of (29) by ‖ +1 ‖ 2 , we obtain , and according to theorem (3.2.1),we see > 0 . Therefor Therefore, new method satisfied the sufficient descent condition.

Global Convergence Analysis for New Method:
In order to prove the global convergence of new nonlinear conjugate gradient methods, under the Wolfe line search. We will impose the following assumptions and lemma about the objective function ℎ( ).

Theorem (3.3.1):
let be generated by new CG-method (14) and function ℎ( ) is uniformly convex, if assumptions (3.3.1) holds then we have Proof: from (14) , we have we know that the +1 < and using triangle and Cauchy-Schwarz inequalities in (37) and we obtain and return to hypothesis (ii), and using ‖ +1 ‖ ≤ γ , we get Taking norm for both sides of new direction in (17): Apply preliminary of triangle inequality in (40), and we get From (38) and (41), we obtain implies that and (47) Based on Powell's Equation for Solving Large-Scale Unconstrained Optimization

Numerical Results
It is important to take note that the theoretical prove it is not adequate to show the effectiveness or robust of any conjugate gradient method. Thus, there would be need to study the method numerical by evaluate the performance method on group of test problems and evaluation the number of iteration or computation time (CPU time).
In this section, is recorded some numerical results by comparing the suggestion method with classical conjugate gradient methods. All of the methods are coded in Fortran95 and stopping the iteration when this statement ‖ +1 ‖ < 10 −5 is true. For testing we used the well-known nonlinear problems with dimension ranging between 4 to 5000 [1], all methods are using cubic fit mothed to find the steplength > 0 under conditions (3) and (4)  In table 1, the results show that the performance of the new method is better than the HS and Perry methods based on the total numbers of function evaluations and its gradient evaluations.
As a final note, table 2 demonstrations the performance of new method when the orthogonal condition is available ⟨ +1 , ⟩ = 0 . During the numerical experiments outcome from comparing between New method and Dai and Yuan method / and see that the new method is top perform to Dai and Yuan method.

Conclusions
In this work, we present a modified Perry's conjugacy coefficient for conjugate gradient method. The proposed new method is apposite for solving large-scale unconstrained optimization problems because of its lower storage requirement. Furthermore, and under suitable conditions we show that the new method is satisfy the descent, sufficient descent conditions and it is global convergent also the new method gives auspicious numerical results.
In future it can be used the new method to train the neural network in order to improve performance and in the image processing.