Matlab Curve Fitting Algorithm

I was trying to solve a surface fitting problem where I had two inputs [X1 X2] used to predict a third quantity Y that occupied the range [1,0). I initially created a very simple gradient descent script from scratch in Python. It was a traditional gradient descent with RMSE as the cost function. After playing aorund with different learning rates and starting guesses (learning along the way that the problem/solution seemed to be extrmeley sensitive to the learning rate and would easily diverge) the best result I was able to get was .05 RMSE.

I tried the matlab fit function next, with 'poly11' fit type and it found a surface with .0045 RMSE (1 order magnitude better than I achieved). It's not surprising to me that Matlab has a more sophisticated curve fitting algorithm than the rudamentary one I wrote up, but does anyone have an idea of what additional tricks fit() may be using that I'm not?

Answer 1

[Matt J]

If you didn't specify any lower/upper bounds in the fitoptions, then the 'poly11' fitting task has the form of an unconstrained linear-least squares problem and has a closed-form, linear algebraic solution. It is likely that no iterations were done by fit() it all, but rather it probably just solved an appropriate system of linear equations.

[AlphaDecay]

If I were to use something like 'poly22' then would some form of gradient descent be used?

[John D'Errico]

No. poly22 would still be a simple linear least squares. Linear here does NOT indicate the order of the model, but in how the parameters appear in the model. Any polynomial of that sort is included in that form.

[AlphaDecay]

Thank you, that makes sense. I have a final follow-on question. Where do the confidence intervals come from in fit()?

[Matt J]

From here, I assume

https://www.mathworks.com/help/releases/R2008a/toolbox/optim/index.html?/help/releases/R2008a/toolbox/optim/ug/f19096.html&https://www.mathworks.com/matlabcentral/answers/465557-calculating-covariance-matrix-from-jacobian-using-lsqcurvefit#bre4quc-1