Changing optimization technique for Gaussian process regression model

By default, GPR model uses 'bayesopt' optimizer to optimize the hyperparameters. I wish to use 'particle swarm optimization'

to optimize the hyperparamaters i.e. to minimize the loss function or the MSE. Please help.

clear;clc;close all

load('data001.mat')

x = data001(:,1);

y = data001(:,2);

rng default

gprMdl = fitrgp(x,y,'KernelFunction','squaredexponential',...

'OptimizeHyperparameters','auto','HyperparameterOptimizationOptions',...

struct('AcquisitionFunctionName','expected-improvement-plus'));

ypred = resubPredict(gprMdl);

figure();

plot(x,y,'r.');

hold on

plot(x,ypred,'k','LineWidth',2);

xlabel('x');

ylabel('y');

hold off

Answer 1

[Alan Weiss]

I answered a similar question recently: https://www.mathworks.com/matlabcentral/answers/1751170-svm-and-knn-hyperparameter

Alan Weiss

MATLAB mathematical toolbox documentation

[Josh]

Thanks for your reply. I tried to modify the code, however, I am getting some error. Please help.

clear;clc;close all

load('data001.mat')

x = data001(:,1);
y = data001(:,2);

% Define some variables to optimize
Hyperparameters = [optimizableVariable('KernelFunction',{'exponential','squaredexponential','matern32','matern52',...
                   'rationalquadratic'}, 'Type','categorical');
                   optimizableVariable('KernelScale', [1, 3]);optimizableVariable('Sigma',[1e-4,10],'Transform','log')];
% Define the objective function 
fobj = @(hparams)objFcn(hparams,x,y);          

results = particleswarm(fobj,3);  %using pso 

% Look at the best hyperparameters found
bestHyperparameters = bestPoint(results)

% Fit a "final" model to the full dataset using the best hyperparameters
FinalModel = fitrgp(x,y,'KernelFunction',char(bestHyperparameters.KernelFunction),...
                        'KernelScale', bestHyperparameters.KernelScale,...
                        'Sigma', bestHyperparameters.Sigma);

% Define a function to fit a GP
function Loss = objFcn(hparams,x,y)

m = fitrgp(x,y,'KernelFunction',string(bestHyperparameters.KernelFunction),...
                        'KernelScale', bestHyperparameters.KernelScale,...
                        'Sigma', bestHyperparameters.Sigma,'Standardize', true,'KFold', 5);
Loss = kfoldLoss(m);
end

[Alan Weiss]

Please show the errors that you get. All the returned messages in red.

Alan Weiss

MATLAB mathematical toolbox documentation

[Josh]

Unable to resolve the name 'bestHyperparameters.KernelFunction'.

Error in solution>objFcn (line 28)

m = fitrgp(x,y,'KernelFunction',string(bestHyperparameters.KernelFunction),...

Error in solution (line 13)

fobj = @(hparams)objFcn(hparams,x,y);

Error in particleswarm>makeState (line 772)

firstFval = objFcn(state.Positions(1,:));

Error in particleswarm>pswcore (line 174)

state = makeState(nvars,lbMatrix,ubMatrix,objFcn,options);

Error in particleswarm (line 151)

[x,fval,exitFlag,output] = pswcore(objFcn,nvars,lbRow,ubRow,output,options);

Caused by:

Failure in initial objective function evaluation. PARTICLESWARM cannot continue.

[Alan Weiss]

You are conflating different methods of optimizing. The particleswarm solver does not use optimizableVariable variables. Only bayesopt uses those variables. If you want to use particleswarm then you have to use variables that particleswarm understands, which are row vectors of standard MATLAB double-precision variables.

In the example I pointed you to, the objective function is

fun = @(x)kfoldloss(fitcsvm(X,Y,'CVPartition',c,'BoxConstraint',x(1),'KernelScale',x(2)));

Now maybe that is too compact, but let's try to understand what it is doing. The x variable has two components, x(1) and x(2). The fitcsvm call sets the BoxConstraint parameter to x(1) and the KernelScale parameter to x(2). The kfoldloss call turns the resulting fit into a measure of loss, meaning the objective function is the loss of the fit with BoxConstraint and KernelScale set to their x values.

So what you need to do is figure out what your variables are that you want to move to reach an optimum, get the function in the form fun(x) where x is a vector of variable values, and then call particleswarm on that function. No optimizeable variables allowed. OK? Copy my function, it works.

Alan Weiss

MATLAB mathematical toolbox documentation

[Josh]

Okay, thanks for the information. There are 4 to 5 variables that needs to be optimized so that that loss function value is minimum. For this example, I wish kernelfunction & kernelscale to be optimized to give minimum value of loss function. I further modified the code, however, errors are still there. It has been days that I am quite stuck with this issue.

Please help to resolve this issue out.

clear;clc;close all

load('data001.mat')

x = data001(:,1);
y = data001(:,2);

KernelFunction = {'exponential','squaredexponential','matern32','matern52','rationalquadratic'};
KernelScale =  [1, 3];

c = cvpartition(y,'KFold',10);
fobj = @(x)kfoldloss(fitrgp(x,y,'CVPartition',c, 'KernelFunction', x(1),'KernelScale',x(2)));

[sol,fval] = particleswarm(fobj,2);

FinalModel = fitrgp(x,y,'KernelFunction', sol(1),'KernelScale',sol(2));

Errors:

Error using classreg.learning.modelparams.GPParams.make

'KernelFunction' value must be a function handle, or a character vector or string scalar containing the name of a MATLAB function on the current

path. The name can also be one of: Exponential, ARDExponential, SquaredExponential, ARDSquaredExponential, Matern32, ARDMatern32, Matern52,

ARDMatern52, RationalQuadratic, ARDRationalQuadratic.

Error in classreg.learning.FitTemplate/fillIfNeeded (line 708)

this.MakeModelParams(this.Type,this.MakeModelInputArgs{:});

Error in classreg.learning.FitTemplate.make (line 140)

temp = fillIfNeeded(temp,type);

Error in classreg.learning.FitTemplate/fillIfNeeded (line 519)

classreg.learning.FitTemplate.make(this.Method,'type',this.Type,...

Error in classreg.learning.FitTemplate.make (line 140)

temp = fillIfNeeded(temp,type);

Error in RegressionGP.fit (line 310)

temp = classreg.learning.FitTemplate.make('GP','type','regression',varargin{:});

Error in fitrgp (line 475)

obj = RegressionGP.fit(X,Y,RemainingArgs{:});

Error in pso_trial>@(x)kfoldloss(fitrgp(x,y,'CVPartition',c,'KernelFunction',x(1),'KernelScale',x(2))) (line 39)

fobj = @(x)kfoldloss(fitrgp(x,y,'CVPartition',c, 'KernelFunction', x(1),'KernelScale',x(2)));

Error in particleswarm>makeState (line 772)

firstFval = objFcn(state.Positions(1,:));

Error in particleswarm>pswcore (line 174)

state = makeState(nvars,lbMatrix,ubMatrix,objFcn,options);

Error in particleswarm (line 151)

[x,fval,exitFlag,output] = pswcore(objFcn,nvars,lbRow,ubRow,output,options);

Error in pso_trial (line 41)

[sol,fval] = particleswarm(fobj,2);

Caused by:

Failure in initial objective function evaluation. PARTICLESWARM cannot continue.

[Alan Weiss]

There are several errors in your setup, but the most important one is this: you are attempting to use particleswarm to choose an integer or categorical variable. But particleswarm is for continuous variables only. It cannot reliably choose a kernel function.

I am truly baffled as to why you are trying so hard to not use the built-in optimizer that is tailor-made for this purpose. But that is your choice.

If you really want to use another optimizer (and I think it is a mistake to do so), you have to use one that allows integer variables. Most likely ga. So you have 5 kernel functions to choose from. Let x(1) be an integer from 1 through 5. You need to map the x(1) variable to the appropriate name, like this:

krnl = KernelFunction {x(1)};

You seem to want to optimize the kernel scale also. According to the documentation of fitrgp, " fitrgp uses the KernelParameters argument to specify the value of the kernel scale parameter, which is held constant during fitting." However, according to the KernelParameters argument documentation, this argument must be "Initial values for the kernel parameters, specified as a vector. " It is just initial values, they are not held constant. So I don't think that you can do what you want to do using ga. Instead, let's optimize the Sigma argument for values between 1e-3 and 10.

Now your data has continuous values for the variables, as you are using regression, not classification. You have to change the cross-validation partition to no stratification.

Your final code is as follows:

load('data001.mat')
X = data001(:,1);
Y = data001(:,2);
KernelFunction = {'exponential','squaredexponential','matern32','matern52','rationalquadratic'};
c = cvpartition(Y,'KFold',5,'Stratify',false);
fobj = @(x)kfoldLoss(fitrgp(X,Y,'CVPartition',c,'KernelFunction',KernelFunction{x(1)},'Sigma',x(2)));
intcon = 1; % intcon is the indicator of integer variables
lb = [1,1e-3]; % set lower bounds
ub = [5,1e1]; % set upper bounds
[sol,fval] = ga(fobj,2,[],[],[],[],lb,ub,[],intcon);
FinalModel = fitrgp(X,Y,'KernelFunction', KernelFunction{sol(1)},'Sigma',sol(2));

I do not understand why you would use this code. I see no advantage over the built-in optimization capabilities of fitrgp. But suit yourself.

Alan Weiss

MATLAB mathematical toolbox documentation

[Josh]

Thanks for being generous with your patience to my questions and elaborating on it with some information.

Appreciate for responding. Thank you.