possible to reduce the relative difference for CheckGradients in optimization (fmincon)?

When double-checking my Jacobian using CheckGradients, I have a relative maximum difference of, crudely, 4e-6,

and my entries of the Jacobian are in the ballpark 1e-1.

I do not expect that Finite Difference can calculate the first six digits after the decimal correctly as my problem has large second derivatives.

That said, can I reduce the relative difference in CheckGradients from 1e-6 to a different value?

Answer 1

[Matt J]

This FEX download allows you to obtain the results of the finite differencing, as well as to decide where (at what x) you want it computed,

https://www.mathworks.com/matlabcentral/fileexchange/13490-adaptive-robust-numerical-differentiation?s_tid=srchtitle

Occasionally also, I find it to be more accurate and reliable as a checking tool than CheckGradient.

In any case, because it gives you the computed FD gradient, you can use this tool to compare both the finite difference and analytical gradients according to whatever error criterion that you wish.

[SA-W]

Thank you, I will check that reference out!

Answer 2

[Torsten]

Why not just turning off the CheckGradient feature if you are sure your derivatives are correct within a certain error tolerance (which might be larger than 1e-6) ?

[SA-W]

This is an option.

But in case there is an error, I could probably say my Jacobian is wrong if the FD Jacobian differs already at the second or third index. That is why I am looking for a way to change it.

[Torsten]

If you know which MATLAB function is called to check the gradients, you should be able to find a "1e-6" in the code :-)

[SA-W]

I will check that.

To me, the fixed value of 1e-6 does not make sense at all. Say, the entries were in the ballpark 1e-6 -- gradient check would work even if the first relevant digit is wrong. On the other hand, if the entries were 1e-1, gradient check fails although the first 4 or 5 digits are correct.

The value of 1e-6 should be somehow scaled to the orders of magnitude inherent in the problem.

[Torsten]

To me, the fixed value of 1e-6 does not make sense at all. Say, the entries were in the ballpark 1e-6 -- gradient check would work even if the first relevant digit is wrong. On the other hand, if the entries were 1e-1, gradient check fails although the first 4 or 5 digits are correct.

I don't think that 1e-6 is an absolute, but a relative error in the calculation. Thus

|| gradient_MATLAB - gradient_USER || <= eps * ||gradient_MATLAB||

or something similar is checked for

||gradient_MATLAB|| > 1

and

|| gradient_MATLAB - gradient_USER|| <= eps

only for

||gradient_MATLAB|| <= 1

[SA-W]

Let me asked one more question regarding your last comment (if you still get notifications about that chat. If not, I will create a new chat).

What you described in your last comment is exactly what is also stated in the documentation.

Example: gradient_USER(1) = 1e-7, gradient_MATLAB(1)=2e-7

|| gradient_MATLAB(1) - gradient_USER(1) || = 1e-7 < 1e-6

This would mean that GradientCheck works, although gradient_USER(1) is, clearly, wrong. Is that true?