How to retrain existing and trained Neural Network without destroying trained content?

Matlab train() function used for training the neural network initializes all weights and other internal parameters of the network at the beginning. I would like to take a trained network and train it further using new set of data without reinitializing and starting from scratch (destroying the trained net basically). How do I do it ? Is there a way to use existing train() function or is there some re-train() function or do I need to build my own custom solution ? The last option seems unlikely, given the amount of the existing Matlab code. Please advise, Thanks, Mirek Wilmer [e-mail]

[manisha milani]

y=net(x) % y is the trained output
mynet = net; 
save mynet;
load mynet;
test = mynet(new_data);

Answer 1

[Greg Heath]

TRAIN initializes weights ONLY IF ALL weights are zero.

OTHERWISE

TRAIN will update weights with the new data.

Therefore old weights will be modified to fit the new data.

If the new data is unlike the old data, performance on the old data will be degraded.

In order to prevent "FORGETTING", a characteristic subset of the old data SHOULD BE MIXED with the new data for the retraining.

Hope this helps.

Thank you for formally accepting my answer

Greg

Answer 2

[Greg Heath]

In order to preserve the dominant characteristics of the 1st dataset you must include that information while adapting to the new data. Therefore I always represent the 1st data set by a representative subset and add that to the new set.

I have been most successful when using Elliptic or Gaussian basis functions (with limited effective range) in the net.

Hope this helps.

Thank you for formally accepting my answer

Greg

[TROY TULLY]

Hi Greg,

Do you know of a way to have a computer search through the training data to find a represntative subset?

Answer 3

[KSSV]

The function train gives out a structure variable net with all the information of training the inputs have under gone.

[net,tr] = train(net,.......)

If you don't clear your workspace, you can use the variable net for what you want. It will retain all the previous data.

[net,tr] = train(net,newdata.....)

[Mirek Wilmer]

but what about train() function initializing the weights ? Wouldn't this destroy ( re-initialize ) trained weights in the net structure variable ?

[Moritz Cremer]

I have the same problem with different time-series datasets.

I can not just connect them to one big dataset like here, because the end of the first dataset and the beginning of the second dataset doesn't match and would for sure confuse the training of the network.

Can I save the weights&biases from the first training and then initialize the net with these weights&bias at the next training iteration?

[Gideon Prior]

That isnt the case unfortunately. Calls to 'train' reinitialize the weights, which is odd since there is a method 'init' already included in the toolbox. I dont know why the mathworks folks decided to not include an option to prevent initialization when 'train' is called.

[Kinga Wilmer]

As a mater of fact, if the workspace is not cleared and the definition of net is not changed, then train() function does not destroy the existing weights. I did trace the whole process with a debugger step by step and confirmed it. Therefore you can re-use your trained net for further training. Mind you, the additional data may re-train your net, so if you want to retain the existing trained capacity, you need to keep statistically important sample representation of your previous training data in the new training dataset. Therefore only small incremental retraining steps make sense. Another way is to train number of separate networks and combine them into hierarchical model.

Answer 4

[kira]

If you want to train incrementally on your data set, you can also do it all at once by dealing with them as concurrent data https://www.mathworks.com/help/deeplearning/ug/multiple-sequences-with-dynamic-neural-networks.html