You are on page 1of 22

# MAE

## 206 - Fall 2017

HOMEWORK #6 Laura Novoa - #52405314
Problem 1 First initial condition

## iter mu epsilon x1 x2 x3 h Parameters:

Steepest Armijo 11 6000 0.0000033333 1.4141302215 0.0000012422 ‐0.0000005145 0.0001178607 Initial condition: mu epsilon
Steepest  freezes at j=254
i<300 Qfit 2 6000 0.0003333333 1.4141301744 0.0003473816 ‐0.0001438600 0.0001178542 1 20*j 0.01/j
Steepest MATLAB 2 6000 0.0000333333 1.4141302216 0.0000148585 ‐0.0000061484 0.0001178581 ‐0.2
BFGS Armijo 3 6000 0.0000333333 1.4141302217 ‐0.0000014903 ‐0.0000035978 0.0001178581
BFGS Qfit 2 6000 0.0000333333 1.4141302217 0.0000016246 ‐0.0000063678 0.0001178581
BFGS MATLAB 2 6000 0.0000333333 1.4141302215 ‐0.0000023540 ‐0.0000039205 0.0001178581

Discussion:  For the first initial condition, all algorithms converge to the local minimizer
*Since we have an increasing mu ‐> infinite and a decreasing tolerance, the convergence guarantee theorem of the Pentaly Funciton assure that every limit point of our sequence is a stationary point. This statement is valid for al other scenarios simulated
Steepest Descent with quadratic fit algorithm got "stuck" in the while loop at j =254 , with a condition similar to the Rosenbrok's banana funciton, where the incremental improvement is so small and the code "freezes
for this, a limit of 300 iterations was implemented so the algorithm could proceed ( i<300) and implementing this measure came a the cost of a reduced accuracy as we can see from the first plot ||xk‐x*|| and second plot ||grad dQ(x)||
h(x) = 0 is not satisfied for all these solutions. However, we get very close to zero and depending on the  optimizaiton problem, this could be a valid approximation (or not!
Armijo methods required more iterations compared to the other algorithms since this is an inexact line search method
The other methods presented good convergence in 2 iterations
Problem 1 Second initial condition
Quad. Penalty
iter mu epsilon x1 x2 x3 h Parameters:
Steepest Armijo 8 6000 0.0000333333 ‐1.4142968870 0.0000000051 ‐0.0000000021 0.000118 Initial cond mu epsilon
Steepest Qfit  2 6000 0.0000333333 ‐1.4142968883 0.0000110362 ‐0.0000038939 0.000118 5 20*j 0.01/j
Steepest MATLAB 2 6000 0.0000333333 ‐1.4142968882 0.0000047556 0.0000020904 0.000118 5
BFGS Armijo 3 6000 0.0000333333 ‐1.4142968883 0.0000004517 0.0000013279 0.000118 5
BFGS Qfit 2 6000 0.0000333333 ‐1.4142968883 0.0000023877 0.0000034491 0.000118
BFGS MATLAB 2 6000 0.0000333333 1.4141302216 0.0000012886 ‐0.0000005323 0.000118

## Discussion: For the second initial condition, all algorithms converge to the global minimizer except for BFGS + MATLAB

h(x) = 0 is not satisfied for all these solutions. However, we get very close to zero and depending on the  optimizaiton problem, this could be a valid approximation (or not!).
Armijo methods required more iterations compared to the other algorithms since this is an inexact line search method
The other methods presented good convergence in 2 iterations
Problem 1 Second initial condition
Quad. Penalty Reduced mu

## iter mu epsilon x1 x2 x3 h Parameters:

Steepest Armijo 4 300 0.0000333333 ‐1.4158772619 0.0000000000 0.0000000000 0.00235421 Initial cond mu epsilon
Steepest Qfit  2 300 0.0000333333 ‐1.4158772919 0.0000011198 ‐0.0000004641 0.002354253 5 j 0.01/j
Steepest MATLAB 2 300 0.0000333333 ‐1.4158772915 0.0000049723 ‐0.0000020602 0.002354252 5
BFGS Armijo 3 300 0.0000333333 ‐1.4158772920 ‐0.0000002262 0.0000007315 0.002354253 5
BFGS Qfit 2 300 0.0000333333 ‐1.4158772914 ‐0.0000003728 0.0000001544 0.002354252
BFGS MATLAB 2 300 0.0000333333 ‐1.4158772923 ‐0.0000010481 0.0000004342 0.002354253

Discussion:  Mu was decreased to mu = j , so it only goes up to 300.
All algorithms converge to the global minimizer
As expected, the accuracy of the final soluton x* is worse compared to using a larger mu.
h(x) = 0 is also not satisfied for all these solutions.
A higher mu improved the convergence (less iterations )of Steepest Descent + Armijo algorithm
Problem 1 Second initial condition
Quad. Penalty Increased mu

## iter mu epsilon x1 x2 x3 h Parameters:

Steepest Armijo 72 15000 0.0000333333 1.4141802283 0.0000257215 ‐0.0000106542 4.71E‐05 Initial cond mu epsilon
Steepest Qfit  2 15000 0.0000333333 ‐1.4142468944 ‐0.0000102178 0.0000069150 4.71E‐05 5 50*j 0.01/j
Steepest MATLAB 2 15000 0.0000333333 ‐1.4142468944 0.0000112961 ‐0.0000054543 4.71E‐05 5
BFGS freees at j=1  Armijo 7 15000 0.0000333333 ‐1.4142468948 0.0000000083 ‐0.0000000055 4.71E‐05 5
BFGS Qfit 2 15000 0.0000333333 1.4141802278 0.0000163519 ‐0.0000060467 4.71E‐05
BFGS MATLAB 2 15000 0.0000333333 ‐1.4142468945 0.0000035746 0.0000050556 4.71E‐05

Discussion:  Mu was increased to mu = 50*j , so it goes up to 15,000
As a result,BFGS + Armijo got ill‐conditoned and froze at
All algorithms converge to the global minimizer except for Steepets Descent + Armijo and BFGS + Qfit
The accuracy of x* is superior if compared to the previous cases that used smaller mu
BFGS methods were also more accurate
h(x) is also closer to zero as compared to previous cases
BFGS + Armijo had to be adjusted because code "froze". For this, the iterations were limited (i<100)
Problem 1 Second initial condition
Quad. Penalty Increased mu
Lower tolerance

## iter mu epsilon x1 x2 x3 h Parameters:

Steepest Armijo 47 15000 0.0000033333 1.4141802278 0.0000025745 ‐0.0000010664 4.71E‐05 Initial cond mu epsilon
Steepest Qfit  2 15000 0.0000033333 ‐1.4142468945 ‐0.0000007622 0.0000003357 4.71E‐05 5 50*j 0.01/10*j
Steepest MATLAB 2 15000 0.0000033333 ‐1.4142468945 0.0000006424 ‐0.0000002719 4.71E‐05 5
BFGS freezes at j=1
i<100 Armijo 3 15000 0.0000033333 ‐1.4142468945 0.0000001204 ‐0.0000004386 4.71E‐05 5
BFGS freezes at j=49
epsilon0 = 0.1 Qfit 2 15000 0.0000333333 1.4141802278 0.0000163519 ‐0.0000060467 4.71E‐05
BFGS MATLAB 2 15000 0.0000033333 ‐1.4142468946 0.0000001674 ‐0.0000000534 4.71E‐05

Discussion:  Tolerance was lowered to epsilon= 0.01/(10*j)
All algorithms converge to the global minimizer except for Steepets Descent + Armijo and BFGS + Qfit
The accuracy of x* is superior if compared to the previos cases that used higher epsilon
All methods were equally accurate
h(x) is very close to zero
BFGS + Armijo and Quadratic fits had to be adjusted because code "froze". For this, the iterations were limited (i<100) and the tolerance was lowered (epsilon0 = 0.1) for each case, respectively
Problem 2 First initial condition
Augmented Lagrangian
iter mu lambda epsilon x1 x2 x3 h Parameters:
Steepest Armijo 1 6000 0.7071187944 0.0000333333 ‐1.4142135623 0.0000111811 ‐0.0000046319 0.0000000001 Initial cond mu epsilon
Steepest Qfit  1 6000 ‐0.707111228 0.0000333333 1.4142135623 0.0000063460 ‐0.0000026272 0.0000000001 1 20*j 0.01/j
Steepest MATLAB 2 6000 ‐0.707117639 0.0000333333 1.4142135594 0.0000283944 ‐0.0000117613 0.0000000038 0.55
BFGS Armijo 2 6000 ‐0.7070842 0.0000333333 1.4142135666 0.0000000459 0.0000001108 0.0000000060 ‐0.2
BFGS freezes at j =144
Qfit 1 6000 ‐0.708675154 0.0033333333 1.4142127843 0.0011008021 ‐0.0008002352 0.0000001743
*epsilon0 = 1
BFGS  MATLAB 2 6000 ‐0.707116129 0.0000333333 1.4142135602 0.0000153577 ‐0.0000010266 0.0000000030

Discussion:  Augmented Lagrangian is a faster method overall and required less iterations compared to Quadratic Penalty
BFGS + Qfit froze at j=144, to fix this issue, the initial tolerance was lowered to 1
For all the cases it converged, the accuracy ||xk ‐ x*|| is comparable or superior if compared to the Quadratic Pentaly method for the same IC and mu
BFGS + armijo showed best accuracy
For all subsequente cases, the constran value h(x) also were very much reduced in comparison to Quadratic Penalty method
Problem 2 Second initial condition
Augmented Lagrangian
iter mu lambda epsilon x1 x2 x3 h Parameters:
Steepest Armijo 139 6000 ‐0.7071198290 0.0000333333 1.4142135621 0.0000282921 ‐0.0000117190 0.0000000000 Initial cond mu epsilon

Steepest freezes at j=44
*0.01/j^(0.1) Qfit  1 6000 0.710133813 0.0056531157 ‐1.4142131153 0.0012871411 ‐0.0005307967 0.0000003369 5 20*j 0.01/j
Steepest MATLAB 94 6000 0.707120843 0.0000333333 ‐1.4142135654 0.0000062742 0.0000025463 0.0000000043 5
BFGS Armijo 1 6000 0.707102777 0.0000333333 ‐1.4142135624 0.0000000289 0.0000000698 0.0000000000 5
BFGS freezes at j=87
*0.01/j^(0.1) Qfit 2 6000 0.70799064 0.0056531157 ‐1.4142136580 0.0003528936 0.0004415282 0.0000002950
BFGS  MATLAB 1 6000 ‐0.708558485 0.0056531157 1.4142135021 0.0001514039 0.0002424347 0.0000000443

Discussion:  For the second IC, the two algorithms that use Qfit froze. To fiz this issue, the tolerance was lowered setting epsilon = 0.01/j^(0.1)
The accuracy of the two methods was lowered as a consequence
For all the cases it converged, the accuracy ||xk ‐ x*|| is comparable or superior if compared to the Quadratic Pentaly method for the same IC and mu
Problem 2 Second initial condition
smaller mu

## iter mu lambda epsilon x1 x2 x3 h Parameters:

Steepest Armijo 1 150 0.7070896073 0.0000333333 ‐1.4142135634 0.0000000000 0.0000000000 0.0000000015 Initial cond mu epsilon
Steepest  Qfit  1 150 0.707086663 0.0000333333 ‐1.4142135609 0.0000000140 0.0000000339 0.0000000021 5 0.5*j 0.01/j
Steepest MATLAB 1 150 0.707114907 0.0000333333 ‐1.4142135630 0.0000000214 0.0000000517 0.0000000008 5
BFGS Armijo 1 150 0.707123296 0.0000333333 ‐1.4142136137 0.0000000185 0.0000000446 0.0000000725 5
BFGS  Qfit 1 150 0.707092724 0.0000333333 ‐1.4142135565 ‐0.0000022125 0.0000008821 0.0000000083
BFGS i<1000 MATLAB 1 150 0.707119342 0.0000333333 ‐1.4142135644 ‐0.0000015075 0.0000006081 0.0000000028

Discussion:  A lower mu = 0.5 got rid of the instabilities and ill conditonings and allowerd for a scenario where all algorithms converge to the global minimun
Problem 2 Second initial condition
smaller mu

## iter mu lambda epsilon x1 x2 x3 h Parameters:

Steepest Armijo 1 300 0.7071294759 0.0000333333 ‐1.4142135837 0.0000000000 0.0000000000 0.0000000301 Initial cond mu epsilon

Steepest freezes at j=59
*0.01/j^(0.1) Qfit  1 300 0.705047867 0.0056531157 ‐1.4142135185 0.0000026049 0.0000058721 0.0000000620 5 j 0.01/j
Steepest MATLAB 1 300 0.707101632 0.0000333333 ‐1.4142135544 0.0000007569 0.0000015836 0.0000000113 5
BFGS Armijo 2 300 0.707101548 0.0000333333 ‐1.4142135387 0.0000000048 0.0000000116 0.0000000335 5
BFGS freezes at j=238
*0.01/j^(0.1) Qfit 1 300 0.710473711 0.0056531157 ‐1.4142136151 0.0001078058 ‐0.0000449355 0.0000000813
BFGS i<1000 MATLAB 1 300 0.707087506 0.0000333333 ‐1.4142135493 0.0000049338 ‐0.0000020659 0.0000000184

Discussion:  However a slight increase in mu ( mu = 1) already makes the system ill conditioned for the algorithm that use Quadratic fit.
Problem 2 Second initial condition
Augmented Lagrangian Larger mu

## iter mu lambda epsilon x1 x2 x3 h Parameters:

Steepest Armijo 1 15000 ‐0.7071153787 0.0000333333 1.4142135620 0.0000312953 ‐0.0000129629 0.0000000000 Initial cond mu epsilon
Steepest freezes at j=39
Qfit  2 15000 0.70907484 0.0056531157 ‐1.4142134220 0.0001731697 0.0009447634 0.0000002627
*0.01/j^(0.1) 5 50*j 0.01/j
Steepest MATLAB 2 15000 0.707094765 0.0000333333 ‐1.4142135612 0.0000114086 ‐0.0000047053 0.0000000016 5
BFGS Armijo 3 15000 0.707106782 0.0000333333 ‐1.4142135613 0.0000000063 0.0000000152 0.0000000015 5
BFGS freezes at j=1
none of the measures  Qfit
made it converge
BFGS  MATLAB 2 15000 0.707101268 0.0000333333 ‐1.4142135618 0.0000018305 0.0000029599 0.0000000007

Discussion:  A high mu causes even more ill‐conditioning, as the codes are "freezing"  in eralier js.
However, the Augmented Lagrangian method does not require a larger mu to produce good reuslts, since we obtained very good reuslts with mu = 0.5*j.
Thus, we can conclude Augmented Lagrangian is a more robust algorithm in terms of convergence
MATALB CODE

PracticalPenalty.m

clear all
close all
clc

%-------------
algorithm_selector=2;
%1 Steepest Descent
%2 Quasi Newton BFGS
Stepsize_selector=2;
%1 Armijo
%2 Use quadratic fit
%3 Use Matlab's search algorithm
%-----Initial Conditions
x1=5%5;%1;
x2=5%5;%0.55;
x3=5%5;%-0.2;

%-----------------
%
epsilon0=0.01; %initial stopping condition tolerance
%epsilon0=0.1; %initial stopping condition tolerance

%%%%%%%%%%%%%%%%%%%%%%%%

for j=1:300
j
mu=j*50;
STOP=1;
H=eye(3,3); %this is the intial condition for Quasi Newton
i=1;
clear g_norm;
These were the new "stopping
%N=500; %number of iterations to solve Q(x,mu)
epsilon=epsilon0/(1*j);%^(0.1);
conditions" implemented in
while (STOP) case the algorithm freezes
g=[1+mu*(x1)*(0.5*x1^2+0.5*x2^2+0.5*x3^2-1);
2*x2+x3+mu*(x2)*(0.5*x1^2+0.5*x2^2+0.5*x3^2-1);
x2+4*x3+mu*(x3)*(0.5*x1^2+0.5*x2^2+0.5*x3^2-1)];
g_norm(i)=norm(g);

if g_norm(i)<epsilon % || i>100
STOP=0;
%saving x* vector, for each mu(j) for plotting convergence in
%the end
xk(:,j) = [x1 ; x2 ; x3]; Saving results to plot
g_norm_j(j)=g_norm(i);
h_j(j) = norm(0.5*x1^2 + 0.5*x2^2 + 0.5*x3^2 - 1);
i=i+1;
else
if algorithm_selector==1
dk=-g; %for steepest descent use
else
dk=-H*g;
end

if Stepsize_selector==1
ff_cost_penalty=@ (p)
(p(1)+p(2)^2+p(2)*p(3)+2*p(3)^2+0.5*mu*(0.5*p(1)^2+0.5*p(2)^2+0.5*p(3)^2-1)^2);
grad_ff_cost_penalty=@(p)([1+mu*(p(1))*(0.5*p(1)^2+0.5*p(2)^2+0.5*p(3)^2-1);
2*p(2)+p(3)+mu*(p(2))*(0.5*p(1)^2+0.5*p(2)^2+0.5*p(3)^2-1);
p(2)+4*p(3)+mu*(p(3))*(0.5*p(1)^2+0.5*p(2)^2+0.5*p(3)^2-1)]);
p=[x1 x2 x3]';
[iterat,alpha_k]=Armijo(ff_cost_penalty,grad_ff_cost_penalty,dk,p,0.5,1e-4);
elseif Stepsize_selector==2
% the quadratic search
[iterat,alpha_k]=quadratic_fit([x1,x2,x3]',dk,mu);
else
%Matlab univariate function minimizer
Mu=mu;
f_cost_penalty=@ (alpha)
((x1+alpha*dk(1))+(x2+alpha*dk(2))^2+(x2+alpha*dk(2))*(x3+alpha*dk(3))+2*(x3+alpha*dk(3))^2+...
0.5*Mu*(0.5*(x1+alpha*dk(1))^2+0.5*(x2+alpha*dk(2))^2+0.5*(x3+alpha*dk(3))^2-
1)^2);
options = optimset('TolX',1e-8);
alpha_k = fminbnd(f_cost_penalty,0,1,options);
end

x1=x1+alpha_k*dk(1);
x2=x2+alpha_k*dk(2);
x3=x3+alpha_k*dk(3);

if algorithm_selector==2
g_kplus1=[1+mu*(x1)*(0.5*x1^2+0.5*x2^2+0.5*x3^2-1);
2*x2+x3+mu*(x2)*(0.5*x1^2+0.5*x2^2+0.5*x3^2-1);
x2+4*x3+mu*(x3)*(0.5*x1^2+0.5*x2^2+0.5*x3^2-1)];
p_k=alpha_k*dk;
q_k=g_kplus1-g;
H=H+(1+(q_k'*H*q_k)/(p_k'*q_k))*p_k*p_k'/(p_k'*q_k)-...
(H*q_k*p_k'+p_k*q_k'*H)/(p_k'*q_k);
end
i=i+1;
end
end
end

x_star=[x1 x2 x3]'
iter=i-1

## %We know in advance the 2 only minimizers

localmin = [sqrt(2) ; 0 ; 0];
globalmin = [-sqrt(2) ; 0 ; 0];

for n=1:j
if x1<0
error(n) = norm(xk(:,n) - globalmin);
else
error(n) = norm(xk(:,n) - localmin);
end
end

## %%%%saving data in file

if algorithm_selector==1 && Stepsize_selector==1
save grad_descent_Armijo.mat g_norm_j mu epsilon iter error h_j;
elseif algorithm_selector==1 && Stepsize_selector==2
save grad_descent_Qfit.mat g_norm_j mu epsilon iter error h_j;
elseif algorithm_selector==1 && Stepsize_selector==3
save grad_descent_MATLAB.mat g_norm_j mu epsilon iter error h_j;
elseif algorithm_selector==2 && Stepsize_selector==1
save BFGS_Armijo.mat g_norm_j mu epsilon iter error h_j;
elseif algorithm_selector==2 && Stepsize_selector==2
save BFGS_Qfit.mat g_norm_j mu epsilon iter error h_j;
elseif algorithm_selector==2 && Stepsize_selector==3
save BFGS_MATLAB.mat g_norm_j mu epsilon iter error h_j;
end

## T = table(iter, mu,epsilon, x_star(1),x_star(2),x_star(3), h_j(j))

T.Properties.VariableNames = {'iterations' 'mu' 'epsilon' 'x1' 'x2' 'x3' 'h'}

T =

## iter mu epsilon Var4 Var5 Var6 Var7

____ _____ __________ ______ __________ ___________ __________

## 2 15000 3.3333e-05 1.4142 1.6352e-05 -6.0467e-06 4.7142e-05

T =

iterations mu epsilon x1 x2 x3 h
__________ _____ __________ ______ __________ ___________ __________

## 2 15000 3.3333e-05 1.4142 1.6352e-05 -6.0467e-06 4.7142e-05

Plotting Results module
figure
load('grad_descent_Armijo.mat')
semilogy([1:1:j],error,'-ks')

hold on
load('grad_descent_Qfit.mat')
semilogy([1:1:j],error,':b')

hold on
load('grad_descent_MATLAB.mat')
semilogy([1:1:j],error,'-.r','MarkerFaceColor','k')

hold on
load('BFGS_Armijo.mat')
semilogy([1:1:j],error,'g')

hold on
load('BFGS_Qfit.mat')
semilogy([1:1:j],error,':m')

hold on
load('BFGS_MATLAB.mat')
semilogy([1:1:j],error,'-.c','MarkerFaceColor','r')

## set(gca, 'YScale', 'log','FontSize', 15)

xlabel('j (\mu_j iteration)')
ylabel('||x_k-x^*||')
legend('Steepest Descent + Armijo','Steepest Descent + Qfit','Steepest Descent + MATALB','BFGS +
Armijo','BFGS + Qfit','BFGS + MATALB','Location','northeast')
axis tight

## Plotting Results gradient(Q)

figure
load('grad_descent_Armijo.mat')
semilogy([1:1:j],g_norm_j,'-ks')

hold on
load('grad_descent_Qfit.mat')
semilogy([1:1:j],g_norm_j,':b')

hold on
load('grad_descent_MATLAB.mat')
semilogy([1:1:j],g_norm_j,'-.r','MarkerFaceColor','k')

hold on
load('BFGS_Armijo.mat')
semilogy([1:1:j],g_norm_j,'g')
hold on
load('BFGS_Qfit.mat')
semilogy([1:1:j],g_norm_j,':m')

hold on
load('BFGS_MATLAB.mat')
semilogy([1:1:j],g_norm_j,'-.c','MarkerFaceColor','r')

## set(gca, 'YScale', 'log','FontSize', 12)

xlabel('j (\mu_j iteration)')
ylabel('||\nabla(Q(x;\mu))||')
legend('Steepest Descent + Armijo','Steepest Descent + Qfit','Steepest Descent + MATLAB','BFGS +
Armijo','BFGS + Qfit','BFGS + MATLAB','Location','northeast')
axis tight

## Plotting Results h(x)

figure
load('grad_descent_Armijo.mat')
plot([1:1:j],h_j,'-ks')

hold on
load('grad_descent_Qfit.mat')
plot([1:1:j],h_j,':b')

hold on
load('grad_descent_MATLAB.mat')
plot([1:1:j],h_j,'-.r','MarkerFaceColor','k')

hold on
load('BFGS_Armijo.mat')
plot([1:1:j],h_j,'g')

hold on
load('BFGS_Qfit.mat')
plot([1:1:j],h_j,':m')

hold on
load('BFGS_MATLAB.mat')
plot([1:1:j],h_j,'-.c','MarkerFaceColor','r')

## set(gca, 'YScale', 'log','FontSize', 12)

xlabel('j (\mu_j iteration)')
ylabel('||h(x)||')
legend('Steepest Descent + Armijo','Steepest Descent + Qfit','Steepest Descent + MATALB','BFGS +
Armijo','BFGS + Qfit','BFGS + MATALB','Location','northeast')

## Published with MATLAB® R2015a

PracticalPenalty_AL.m

clear all
close all
clc

%-------------
algorithm_selector=2;
%1 Steepest Descent
%2 Quasi Newton BFGS
Stepsize_selector=2;
%1 Armijo
%2 Use quadratic fit
%3 Use Matlab's search algorithm
%-----Initial Conditions
x1=5%5;%1;
x2=5%5;%0.55;
x3=5%5;%-0.2;

%-----------------
%
epsilon0=0.01; %initial stopping condition tolerance
%epsilon0=1; %initial stopping condition tolerance
lambda = 1; %initial lambda

%%%%%%%%%%%%%%%%%%%%%%%%

for j=1:300
j
mu=0.5*j;%1*j;
STOP=1;
H=eye(3,3); %this is the intial condition for Quasi Newton
i=1;
clear g_norm;
%N=500; %number of iterations to solve Q(x,mu)
epsilon=epsilon0/(1*j);%^(0.1);
while (STOP)
g=[1+mu*(x1)*(0.5*x1^2+0.5*x2^2+0.5*x3^2-1) + lambda*(x1);
2*x2+x3+mu*(x2)*(0.5*x1^2+0.5*x2^2+0.5*x3^2-1)+ lambda*(x2) ;
x2+4*x3+mu*(x3)*(0.5*x1^2+0.5*x2^2+0.5*x3^2-1)+ lambda*(x3)];
g_norm(i)=norm(g);

if g_norm(i)<epsilon % || i<1000
STOP=0;
%saving x* vector, for each mu(j) for plotting convergence in
%the end
xk(:,j) = [x1 ; x2 ; x3];
g_norm_j(j)=g_norm(i);
h_j(j) = norm(0.5*x1^2 + 0.5*x2^2 + 0.5*x3^2 - 1);
i=i+1;
else
if algorithm_selector==1
dk=-g; %for steepest descent use
else
dk=-H*g;
end

if Stepsize_selector==1
ff_cost_penalty=@ (p)
(p(1)+p(2)^2+p(2)*p(3)+2*p(3)^2+0.5*mu*(0.5*p(1)^2+0.5*p(2)^2+0.5*p(3)^2-
1)^2)+lambda*(0.5*p(1)^2+0.5*p(2)^2+0.5*p(3)^2-1) ;
grad_ff_cost_penalty=@(p)([1+mu*(p(1))*(0.5*p(1)^2+0.5*p(2)^2+0.5*p(3)^2-
1)+lambda*p(1);
2*p(2)+p(3)+mu*(p(2))*(0.5*p(1)^2+0.5*p(2)^2+0.5*p(3)^2-1)+lambda*p(2);
p(2)+4*p(3)+mu*(p(3))*(0.5*p(1)^2+0.5*p(2)^2+0.5*p(3)^2-1)+lambda*p(3)]);
p=[x1 x2 x3]';
[iterat,alpha_k]=Armijo(ff_cost_penalty,grad_ff_cost_penalty,dk,p,0.5,1e-4);
elseif Stepsize_selector==2
% the quadratic search
[iterat,alpha_k]=quadratic_fit_AL([x1,x2,x3]',dk,mu,lambda);
else The function gradient needed to be
%Matlab univariate function minimizer
Mu=mu;
modified t and included the input
f_cost_penalty=@ (alpha) argument "lambda"
((x1+alpha*dk(1))+(x2+alpha*dk(2))^2+(x2+alpha*dk(2))*(x3+alpha*dk(3))+2*(x3+alpha*dk(3))^2+...
0.5*Mu*(0.5*(x1+alpha*dk(1))^2+0.5*(x2+alpha*dk(2))^2+0.5*(x3+alpha*dk(3))^2-
1)^2) + lambda*(0.5*(x1+alpha*dk(1))^2+0.5*(x2+alpha*dk(2))^2+0.5*(x3+alpha*dk(3))^2-1);
options = optimset('TolX',1e-8);
alpha_k = fminbnd(f_cost_penalty,0,1,options);
end

x1=x1+alpha_k*dk(1);
x2=x2+alpha_k*dk(2);
x3=x3+alpha_k*dk(3);

if algorithm_selector==2
g_kplus1=[1+mu*(x1)*(0.5*x1^2+0.5*x2^2+0.5*x3^2-1) + lambda*(x1);
2*x2+x3+mu*(x2)*(0.5*x1^2+0.5*x2^2+0.5*x3^2-1) + lambda*(x2);
x2+4*x3+mu*(x3)*(0.5*x1^2+0.5*x2^2+0.5*x3^2-1)+ lambda*(x3)];
p_k=alpha_k*dk;
q_k=g_kplus1-g;
H=H+(1+(q_k'*H*q_k)/(p_k'*q_k))*p_k*p_k'/(p_k'*q_k)-...
(H*q_k*p_k'+p_k*q_k'*H)/(p_k'*q_k);
end
i=i+1;
end
end
lambda = lambda + mu*((0.5*x1^2+0.5*x2^2+0.5*x3^2)-1);
end

x_star=[x1 x2 x3]'
iter=i-1

## %We know in advance the 2 only minimizers

localmin = [sqrt(2) ; 0 ; 0];
globalmin = [-sqrt(2) ; 0 ; 0];

for n=1:j
if x1<0
error(n) = norm(xk(:,n) - globalmin);
else
error(n) = norm(xk(:,n) - localmin);
end
end

## %%%%saving data in file

if algorithm_selector==1 && Stepsize_selector==1
save grad_descent_Armijo.mat g_norm_j mu epsilon iter error h_j;
elseif algorithm_selector==1 && Stepsize_selector==2
save grad_descent_Qfit.mat g_norm_j mu epsilon iter error h_j;
elseif algorithm_selector==1 && Stepsize_selector==3
save grad_descent_MATLAB.mat g_norm_j mu epsilon iter error h_j;
elseif algorithm_selector==2 && Stepsize_selector==1
save BFGS_Armijo.mat g_norm_j mu epsilon iter error h_j;
elseif algorithm_selector==2 && Stepsize_selector==2
save BFGS_Qfit.mat g_norm_j mu epsilon iter error h_j;
elseif algorithm_selector==2 && Stepsize_selector==3
save BFGS_MATLAB.mat g_norm_j mu epsilon iter error h_j;
end

## T = table(iter, mu,lambda,epsilon, x_star(1),x_star(2),x_star(3), h_j(j))

T.Properties.VariableNames = {'iterations' 'mu' 'lambda' 'epsilon' 'x1' 'x2' 'x3' 'h'}

T =

## iter mu lambda epsilon Var5 Var6 Var7 Var8

____ ___ _______ __________ _______ ___________ __________ __________

T =

## iterations mu lambda epsilon x1 x2 x3 h

__________ ___ _______ __________ _______ ___________ __________
__________

## 1 150 0.70709 3.3333e-05 -1.4142 -2.2125e-06 8.8205e-07

8.3015e-09
Plotting Results module
figure
load('grad_descent_Armijo.mat')
semilogy([1:1:j],error,'-ks')

hold on
load('grad_descent_Qfit.mat')
semilogy([1:1:j],error,':b')

hold on
load('grad_descent_MATLAB.mat')
semilogy([1:1:j],error,'-.r','MarkerFaceColor','k')

hold on
load('BFGS_Armijo.mat')
semilogy([1:1:j],error,'g')

hold on
load('BFGS_Qfit.mat')
semilogy([1:1:j],error,':m')

hold on
load('BFGS_MATLAB.mat')
semilogy([1:1:j],error,'-.c','MarkerFaceColor','r')

## set(gca, 'YScale', 'log','FontSize', 15)

xlabel('j (\mu_j iteration)')
ylabel('||x_k-x^*||')
legend('Steepest Descent + Armijo','Steepest Descent + Qfit','Steepest Descent + MATALB','BFGS +
Armijo','BFGS + Qfit','BFGS + MATALB','Location','northeast')
axis tight

## Plotting Results gradient(Q)

figure
load('grad_descent_Armijo.mat')
semilogy([1:1:j],g_norm_j,'-ks')

hold on
load('grad_descent_Qfit.mat')
semilogy([1:1:j],g_norm_j,':b')

hold on
load('grad_descent_MATLAB.mat')
semilogy([1:1:j],g_norm_j,'-.r','MarkerFaceColor','k')

hold on
load('BFGS_Armijo.mat')
semilogy([1:1:j],g_norm_j,'g')
hold on
load('BFGS_Qfit.mat')
semilogy([1:1:j],g_norm_j,':m')

hold on
load('BFGS_MATLAB.mat')
semilogy([1:1:j],g_norm_j,'-.c','MarkerFaceColor','r')

## set(gca, 'YScale', 'log','FontSize', 12)

xlabel('j (\mu_j iteration)')
ylabel('||\nabla(Q(x;\mu))||')
legend('Steepest Descent + Armijo','Steepest Descent + Qfit','Steepest Descent + MATLAB','BFGS +
Armijo','BFGS + Qfit','BFGS + MATLAB','Location','northeast')
axis tight

## Plotting Results h(x)

figure
load('grad_descent_Armijo.mat')
plot([1:1:j],h_j,'-ks')

hold on
load('grad_descent_Qfit.mat')
plot([1:1:j],h_j,':b')

hold on
load('grad_descent_MATLAB.mat')
plot([1:1:j],h_j,'-.r','MarkerFaceColor','k')

hold on
load('BFGS_Armijo.mat')
plot([1:1:j],h_j,'g')

hold on
load('BFGS_Qfit.mat')
plot([1:1:j],h_j,':m')

hold on
load('BFGS_MATLAB.mat')
plot([1:1:j],h_j,'-.c','MarkerFaceColor','r')

## set(gca, 'YScale', 'log','FontSize', 12)

xlabel('j (\mu_j iteration)')
ylabel('||h(x)||')
legend('Steepest Descent + Armijo','Steepest Descent + Qfit','Steepest Descent + MATALB','BFGS +
Armijo','BFGS + Qfit','BFGS + MATALB','Location','northeast')

## Published with MATLAB® R2015a

quadratic_fit_AL.m
function [iterat,a_k]=quadratic_fit(xk,dk,mu,lambda)
%starting points [-3 0 1 ]
a(1,1)=0;
a(1,2)=0.5;
a(1,3)=1;
x1=xk(1);
x2=xk(2);
x3=xk(3);
f_cost_penalty=@ (alpha)
((x1+alpha*dk(1))+(x2+alpha*dk(2))^2+(x2+alpha*dk(2))*(x3+alpha*dk(3))+2*(x3+alpha*dk(3))^2+...
0.5*mu*(0.5*(x1+alpha*dk(1))^2+0.5*(x2+alpha*dk(2))^2+0.5*(x3+alpha*dk(3))^2-
1)^2) + lambda*(0.5*(x1+alpha*dk(1))^2+0.5*(x2+alpha*dk(2))^2+0.5*(x3+alpha*dk(3))^2-1);

f(1,1)=f_cost_penalty(a(1,1));
f(1,2)=f_cost_penalty(a(1,2));
f(1,3)=f_cost_penalty(a(1,3));
a(1,4)=0;%this is just a place holder
N=1000;
flag_break=0;
for i=1:N
f(i,1)=f_cost_penalty(a(i,1));
f(i,2)=f_cost_penalty(a(i,2));
f(i,3)=f_cost_penalty(a(i,3));
a12=a(i,1)-a(i,2);
a31=a(i,3)-a(i,1);
a23=a(i,2)-a(i,3);
b12=a(i,1)^2-a(i,2)^2;
b31=a(i,3)^2-a(i,1)^2;
b23=a(i,2)^2-a(i,3)^2;
a(i,4)=0.5*(b23*f(i,1)+b31*f(i,2)+b12*f(i,3))/(a23*f(i,1)+a31*f(i,2)+a12*f(i,3));
a4=a(i,4);
f(i,4)=f_cost_penalty(a4);
delta=1e-10;
if abs(a4-a(i,1))<delta
a(i,4)=a(i,1)+delta;
end
if abs(a4-a(i,3))<delta
a(i,4)=a(i,3)-delta;
end
if (abs(a4-a(i,2))<delta && a(i,2)>0.5*(a(i,1)+a(i,3)))
a(i,4)=a(i,2)-delta;
end
if (abs(a4-a(i,2))<delta && a(i,2)<=0.5*(a(i,1)+a(i,3)))
a(i,4)=a(i,2)+delta;
end

## if (a(i,4)>=a(i,2) && f(i,4)>=f(i,2))

a(i+1,1)=a(i,1);
a(i+1,2)=a(i,2);
a(i+1,3)=a(i,4);
elseif (a(i,4)>=a(i,2) && f(i,4)<f(i,2))
a(i+1,1)=a(i,2);
a(i+1,2)=a(i,4);
a(i+1,3)=a(i,3);
elseif (a(i,4)<a(i,2) && f(i,4)>=f(i,2))
a(i+1,1)=a(i,4);
a(i+1,2)=a(i,2);
a(i+1,3)=a(i,3);
else
a(i+1,1)=a(i,1);
a(i+1,2)=a(i,4);
a(i+1,3)=a(i,2);
end
if abs(a(i+1,3)-a(i+1,1))<10^(-5) %if threshold is satisfied it breaks the for loop and
stops
[fs,index]=min([f(i,1) f(i,2) f(i,3) f(i,4)]);
a_k=a(i,index); %the minimum value
flag_break=1;
iterat=i;
break
end
end
if flag_break==0
[fs,index]=min([f(end,1) f(end,2) f(end,3) f(end,4)]);
a_k=a(end,index); %the minimum value
iterat=N;
end

%plot(a_k,fs,'+')