Newton method to optimize function, error

Hey guys,

im trying to use the newton method for optimizing my following function. I also used the Grid-search method, this worked totaly fine.

Now here is my try with the Newton method:

Imax=100; % Anzahl an maximalen Iterationen
tol=1;
i=1; % Zähler der Schleife
f=[diff(logL,b);diff(logL,y);diff(logL,A);diff(logL,B)]; % Partielle Ableitungen von logL
J=[diff(f,b) diff(f,y) diff(f,A) diff(f,B)]; % Aufstellen der Jacobi-Matrix

X=zeros(Imax,length(J)); % Matrix erstellen
X(1,:)= [27 3 0.04 0.3]; % Startwerte für 1.Iteration
while tol>=1e-9 % Toleranz
% Einsetzen der Werte für Startwertvariablen in Gleichung
fvar=subs(f,{'b' 'y' 'A' 'B'},{X(i,1) X(i,2) X(i,3) X(i,4)});
f_var=subs(J,{'b' 'y' 'A' 'B'},{X(i,1) X(i,2) X(i,3) X(i,4)});
% Evaluieren für Werte für Startwertvariablen
fvalue=eval(fvar);
f_value=eval(f_var);
if max(max(isnan(f_value))) == 1
break
end
% Neuer X Lösungsvektor
*| |*X(i+1,:)=([X(i,1) X(i,2) X(i,3) X(i,4)]'-f_value\fvalue)';*||*
% Abweichung zwischen neuen und alten Werten
tol=abs([X(i,1) X(i,2) X(i,3) X(i,4)] - [X(i+1,1) X(i+1,2) X(i+1,3) X(i+1,4)]);
if i>Imax % Abbruchkriterium
break
end
i=i+1;
end
fprintf('Endergebnis : '); % Endergebnis
[X(i,1) X(i,2) X(i,3) X(i,4)]

Im getting the error message:

> In zwei_parametrig_ohneAB_NEWTON_VERFAHREN (line 54) 
Warning: Matrix is close to singular or badly scaled. Results may be inaccurate. RCOND =  4.698726e-31. 
> In zwei_parametrig_ohneAB_NEWTON_VERFAHREN (line 54) 
Warning: Matrix is close to singular or badly scaled. Results may be inaccurate. RCOND =  7.517962e-30. 
> In zwei_parametrig_ohneAB_NEWTON_VERFAHREN (line 54) 
Warning: Matrix is close to singular or badly scaled. Results may be inaccurate. RCOND =  1.202871e-28. 
> In zwei_parametrig_ohneAB_NEWTON_VERFAHREN (line 54) 
Warning: Matrix is close to singular or badly scaled. Results may be inaccurate. RCOND =  1.924598e-27. 
> In zwei_parametrig_ohneAB_NEWTON_VERFAHREN (line 54) 
Warning: Matrix is close to singular or badly scaled. Results may be inaccurate. RCOND =  3.079253e-26. 
> In zwei_parametrig_ohneAB_NEWTON_VERFAHREN (line 54) 
Warning: Matrix is close to singular or badly scaled. Results may be inaccurate. RCOND =  4.926967e-25.

And after that, MATLAB gives me "wrong" solutions:

 -0.0000 -102.4938  108.4393 -217.2624

By googling the error message i know, that there must be something wrong with my matrix i have marked in the code below. Where is the problem?

Your sincerly, Ronald Singer

[Jan]

Note: "any(isnan(f_value(:)))" is nicer than "max(max(isnan(f_value))) == 1".

Answer 1

[Matt J]

It would help to know what logL looks like.

The message means that your Hessian matrices f_value are close to singular and therefore difficult to invert. You should examine them when the warning is triggered,

>> dbstop if warning

Possibly, you have landed in a region where logL is approximately flat everywhere making the Hessian approximately equal to zero(4). I also notice that you are not doing any line search steps. For non-quadratic function minimization, line searches are needed to assure convergence of Newton's method.

[Ronald Singer]

Hey Matt,

thank you for your answer. LogL looks like:

logL=0;
for i=1:length(Versuchsdaten)
  logL=logL+log(b/(exp((y+A*x1(i)+B*x2(i)))))+(b-1)*log((Versuchsdaten(i))/(exp((y+A*x1(i)+B*x2(i)))))-((Versuchsdaten(i))/exp((y+A*x1(i)+B*x2(i))))^(b);
end

[Ronald Singer]

I don't really know what you mean with line searches...

[Ronald Singer]

I looked at my script and Wikipedia now once again, and i don't really know what line searches means.

Is it possible that it has to be fvalue/f_value and not f_value/fvalue?

[Matt J]

I looked at my script and Wikipedia now once again, and i don't really know what line searches means.

https://en.wikipedia.org/wiki/Line_search

Is it possible that it has to be fvalue/f_value and not f_value/fvalue?

No.

[Ronald Singer]

okay, so you mean point 3 and 4 in this Wikipedia article? i think everthing else is my script doing.

[Matt J]

No, I mean point 4 and 5. You have implemented step 3 by computing the search direction p = f_value\fvalue

[Matt J]

It's also possible that you've chosen a bad initial guess. All zeros

X=zeros(Imax,length(J));

appears to be a shot-in-the-dark selection.

[Ronald Singer]

Sry if i interpret your comment wrong, but the zeros are not my initial guess. My initial guess is:

X(1,:)= [27 3 0.04 0.3]

I've tried to set the initial guess to

X(1,:)= [27.9 3.63 0.004 0.35]

and got the answer i searched for:

ans =

   27.8556    3.6298    0.0046    0.3512

but i mean; i can't guess the solution that precisely, if I don't know it.

[Ronald Singer]

Is it possible to tell Matlab, in which Range of each parameter it should search? I tried it with assume, but it won't listen to me :P

[Matt J]

Not with regular Newton's method. You can do it with fmincon() and other Optimization Toolbox solvers.