Is there a way to train a network giving inputs one by one in MATLAB?

Hi everyone,

I am training a recurrent neural network, and I have to give the data one by one to the network. This is neccessary due to my research. What I wonder is, giving the input one by one decreases the performance of the training. To test this, I created a network and tried to train the network in both ways. I want to show you the results.

%% First Part

rng(1);

numFeatures = 6;

numHiddenUnits = 500;

numResponses = 1;

emir = [ ...

sequenceInputLayer(numFeatures)

lstmLayer(numHiddenUnits,'OutputMode','sequence','InputWeightsInitializer','glorot')

fullyConnectedLayer(numResponses,'WeightsInitializer','glorot')

regressionLayer];

%

%

maxEpochs = 1;

miniBatchSize = 1;

options = trainingOptions('adam', ...

'MaxEpochs',1, ...

'MiniBatchSize',miniBatchSize, ...

'InitialLearnRate',0.01, ...

'GradientThreshold',1, ...

'Plots','training-progress', ...

'Verbose',0);

[emir,info] = trainNetwork(in_norm_arranged ,f_norm_arranged ,emir ,options);

%% Second Part

TrainingLoss = [];

TrainingRMSE = [];

emir = [ ...

sequenceInputLayer(numFeatures)

lstmLayer(numHiddenUnits,'OutputMode','sequence','InputWeightsInitializer','glorot')

fullyConnectedLayer(numResponses,'WeightsInitializer','glorot')

regressionLayer];

%

%

maxEpochs = 1;

miniBatchSize = 1;

options = trainingOptions('adam', ...

'MaxEpochs',1, ...

'MiniBatchSize',miniBatchSize, ...

'InitialLearnRate',0.01, ...

'GradientThreshold',1, ...

'Verbose',0);

[emir,info] = trainNetwork(in_norm_arranged{1} ,f_norm_arranged{1} ,emir ,options);

for i=2:10000

[emir,info] = trainNetwork(in_norm_arranged{i} ,f_norm_arranged{i} ,emir.Layers ,options);

TrainingLoss = [TrainingLoss info.TrainingLoss];

TrainingRMSE = [TrainingRMSE info.TrainingRMSE];

if i == 100;

100

end

end

I stopped the training in 100th iteration to show the difference between these two;

The first image is when I don't use for loop. But the minibatch size is 1, so the input is given one by one.

This figure is when I use for loop and give each input one by one.

I think in for loop, the same ''options'' are used in training, but in the other one some parameters of options are changing throughout training.

Is there a way to train a network giving inputs one by one in MATLAB?

Thanks!

Emirhan

Answer 1

[Ben]

One thing that is changing for sure is the parameters for the ADAM optimizer. For example if you look at adamupdate (doc page), which is the underlying algorithm for the ADAM optimizer, you'll see it has parameters for the average gradients, average squared gradients, and the current iteration. Those are changing throughout training, but when you call trainNetwork each time it has no way to know the values from the previous call to trainNetwork.

A workaround would be to write a custom training loop (doc example) - in custom training you have to manage the optimizer parameters yourself, so you can ensure the two training loops are comparable. In fact you can re-implement most of the training loop from your first part with trainNetwork using a custom training loop - though there are a number of training options you have to write too, see this page for tips.