How to find a polynomial that best fit with scatter plot?

Hi everyone,

I am trying to find a best-fit polynomial equation that works well for my data set (two parameters x, and y).

First, I attempt to fit a polynomial line using Excel (figure attached). The equation shows the best fit with the original scatter plot (the red line is the best fit polynomial line).

However, when I try to estimate values with the best-fit equation, it looks very different from actual data points.

A similar error is observed when I attempt to use the Matlab function polyfilt.

clear all
clc

data=readmatrix('lv_vl.csv');

x=data(:,1);
y=data(:,2);

scatter(x,y,'b')

p = polyfit(x,y,2)

May someone suggets me how i can fix this.

(data is also attached).

Answer 1

[Walter Roberson]

data=readmatrix('lv_vl.csv');
x=data(:,1);
y=data(:,2);
scatter(x,y,5,'.r')

p = polyfit(x,y,2);
eqn = poly2sym(p)

eqn = 

xest = linspace(min(x), max(x), 75);
yest = polyval(p, xest);

hold on
plot(xest, yest, '-b');
title(char(vpa(eqn,5)))

[Andi]

@Walter Roberson Thanks for the help, but still the exact values are a bit farfrom the best-fit polynomial.

for example when x=1500

Actual y is = 3838000

Estimated y=3800000

[Walter Roberson]

You can do slightly better using scaling and centering. But you should expect this to happen in any situation where the data to be fit is not perfectly described by a polynomial of the degree being used.

format long g
data=readmatrix('lv_vl.csv');
x=data(:,1);
y=data(:,2);
scatter(x,y,5,'.r')

[p, s, mu] = polyfit(x,y,2);
eqn = poly2sym(p)

eqn = 

xest = linspace(min(x), max(x), 75);
yest = polyval(p, xest, s, mu);

hold on
plot(xest, yest, '-b');
title(string(vpa(eqn,5)) + " (after scaling and centering)")

y1500 = y(find(x==1500))

y1500 = 
     3838000

y1500est = polyval(p, 1500, s, mu)

y1500est = 
          3809799.11969661

[Andi]

@Walter Roberson While implementing your proposed strategy, I encounter another issue which is overestimation when x becomes lower than a certain value. For example, the best polynomial works when for 2020, 2010, and 1990, but it overestimates the dependent parameter for 2000 and 1980 (when x <1380 it shows an increase in y). One possible way is may interpolate the actual data first and then set a polynomial, but I am not very sure. Looking forward to your suggestion. Thank you!

[Walter Roberson]

format long g
x = 1400:1550;
ytrue = 150*x.^2 - 40000*x + 2.5e8;

p = polyfit(x, ytrue, 2)

p = 1×3

1.0e+00 * 150.000000000001 -40000.0000000033 250000000.000002

ytrue_est = polyval(p, x);

scatter(x, ytrue, '.k');
hold on
scatter(x, ytrue_est, 'or')

ynoise = ytrue + (2 * rand(size(ytrue)) - 1);  %up to +/-1 
pnoise = polyfit(x, ynoise, 2)

pnoise = 1×3

1.0e+00 * 149.999993542975 -39999.9832693142 249999989.346063

ynoise_est = polyval(pnoise, x);
figure
scatter(x, ynoise, '.k');
hold on
scatter(x, ynoise_est, 'or')

figure
plot(x, ytrue-ynoise_est)

max((ytrue - ynoise)./ytrue)

ans = 
      1.94549710870535e-09

So noise only about 2 parts in a billion can make an estimation difference of 2 parts in 2.

But we know from your previous attempts that your data is not actually quadratic -- only close to quadratic. You should expect that it is not going to perfectly match a quadratic estimate.

[Andi]

True, I agree with you, seems interpolation many not work for data limitation. ​ ​

Thank you!

[Walter Roberson]

When I use the Curve Fitting Toolbox to fit your data, the Sum of Squared Error is about 10^11 with polynomial of degree 2. You have to fit somewhere around degree 8 or higher to reduce the error substantially.

[Andi]

I used a degree 4 polynomial to fit my data, which shows very comparable​ results with low error.

[Walter Roberson]

The sum of squared error is still pretty high for degree 4, and there are still fairly visible artifacts.

[Andi]

True, but if I overestimate, then I am unable to incorporate other parameters that have a time-varying influence on the time series. Therefore, I decided to go with a 4-order polynomial.