How to do bayesopt for neural network with unknown layer and neuron number?

%%Choose Variables to Optimize
%hiddenLayerSize, number of neurons
minHiddenLayerSize = 10;
maxHiddenLayerSize = 20;
hiddenLayerSizeRange = [minHiddenLayerSize maxHiddenLayerSize];

%number of hidden layers
minHiddenLayerNumber = 1;
maxHiddenLayerNumber = 3;
hiddenLayerNumberRange = [minHiddenLayerNumber maxHiddenLayerNumber];

optimVars = [
    optimizableVariable('Layer1Size',hiddenLayerSizeRange,'Type','integer')
    optimizableVariable('Layer2Size',hiddenLayerSizeRange,'Type','integer')];

%%Perform Bayesian Optimization
ObjFcn = makeObjFcn();
BayesObject = bayesopt(ObjFcn,optimVars,...
    'MaxObj',30,...
    'MaxTime',8*60*60,...
    'IsObjectiveDeterministic',false,...
    'UseParallel',false);

%%Evaluate Final Network
bestIdx = BayesObject.IndexOfMinimumTrace(end);
fileName = BayesObject.UserDataTrace{bestIdx};
load(fileName);
YPredicted = net(simplefitInputs);
testError = perform(net,simplefitTargets,YPredicted);
testError
valError

% Solve an Input-Output Fitting problem with a Neural Network
% Script generated by Neural Fitting app
% Created 26-Apr-2020 11:07:28
%
% This script assumes these variables are defined:
%
%   Einleger_Binaer_Alle_inv_sortiert - input data.
%   Auslenkung_Alle_inv - target data.

x = Einleger_Binaer_Alle_inv_Sortiert;
t = Auslenkung_Alle_inv;

rng('default');

% Choose a Training Function
% For a list of all training functions type: help nntrain
% 'trainlm' is usually fastest.
% 'trainbr' takes longer but may be better for challenging problems.
% 'trainscg' uses less memory. Suitable in low memory situations.
trainFcn = 'trainlm';  % nicht Levenberg-Marquardt backpropagation, da schneller

% Create a Fitting Network
% hiddenLayerSize = 10;
% net_hiddenlayersize6_sortiert = fitnet(hiddenLayerSize,trainFcn);

% For more hiddenlayers with unknown number of neurons
hiddenLayer1Size = optVars.Layer1Size; 
hiddenLayer2Size = optVars.Layer2Size;
hiddenLayerSizes = [hiddenLayer1Size hiddenLayer2Size];
net = fitnet([hiddenLayer1Size hiddenLayer2Size],  trainFcn);

Hi, I am trying to do a bayes optimization for hyperparameter tuning. I am trying to optimize the number of neurons (Layer Size) and the number of hidden layers. I defined those two variables at the beginning.

How should I construct the layers, if the number of hidden layers and neurons are to be defined and optimized in the bayesopt function? I am kinda confused. Both numbers of neurons and hidden layers are uncertain and should be optimized.

with best regards

Answer 1

[Shivam Singh]

Hello Jingyuan,

It is my understanding that you want to find the optimum number of hidden layers and their respective sizes simultaneously, using Bayesian optimization method with the use of bayesopt function.

You can find a modified example for your case in the live editor file attached.

Describing the Solution:

You can optimize the layer sizes and the number of layers simultaneously. The key is to use only the layers up to the value specified in "LayerNumber".

hiddenLayerSizes = [hiddenLayer1Size hiddenLayer2Size hiddenLayer3Size];  
hiddenLayerSizes = hiddenLayerSizes(1:optVars.LayerNumber);

You can also make use of "ConditionalVariableFcn" name-value pair argument in bayesopt. This function can be used to invalidate unnecessary layers and set them to NaN.

function optVars = condVarFcn(optVars)
   optVars.Layer2Size(optVars.LayerNumber < 2) = NaN;
   optVars.Layer3Size(optVars.LayerNumber < 3) = NaN;
end

Alternate Readymade approach:

If you have Statistics and Machine Learning Toolbox, you can use fitrnet function. There is a built-in Bayesian optimization for this function and is intended to address the exact problem you are facing. By default, you can optimize up to three layers and 300 neurons each. You can also modify this search range and go up to 5 layers and any custom neuron size for each layer. See the examples below.

Minimize Cross-Validation Error in Neural Network

Custom Hyperparameter Optimization in Neural Network

Note that there is only a single solver at this time "LBFGS" and you cannot change it to "trainlm". If you need a variety of train functions and networks, you can customize the attached live script and use the functionality in Deep Learning Toolbox.

I hope this helps you.

[Giuseppe Maria Piuma]

Hello, is there a way to go a step further and find the optimal Transfer Functions (eg logsig,tansig,poslin etc) for each layer?

Thank you in advance