Save intermediate model in matlab while training a deep learning model and resume training from that stage later
11 views (last 30 days)
Show older comments
parvathy prathap on 12 Oct 2021
Commented: Jeet Agrawal on 13 Apr 2023
I am training a 3 pipeline deep learning model in matlab which takes a lot of time to train. I need to store intermediate variable values while training, stop the training process and then resume training at a later point from the stage at which the training was stopped previously. Does Matlab have any options to do this? Any help in this regard would be highly appreciated.
Thanks in Advance
Mahesh Taparia on 15 Oct 2021
You can set the checkpoint path in trainingOptions as suggested in the above answer. The trained weights will be saved into the specified path as a mat file. To resume the training process, you can load those weights in the net variable and start the training process. For example, you can refer this documentation.
Hope it will help!
Jeet Agrawal on 13 Apr 2023
Can you help me understanding load model more.
Do i need to create another file and load model and train from their?
suppose i am saving model after 50 iteration using
checkpointfrequency =50 and checkpointunit = iteration
lastly it has saved checkpoint at 750 now how to run again from 751?
As per link i can reduce my epoch but how can i reduce iteration?
suppose my data set has 20000 files and minibatchsize is 40 for one epoch it will take 500 iteration. I am not able to wait for 500 iteration to complete because it will take around 5 hours. so, want to save at 50 epochs only.
could you please share snipet?
More Answers (1)
yanqi liu on 15 Oct 2021
sir，may be use CheckpointPath，such as
options = trainingOptions('sgdm', ...
'MaxEpochs', 5, ...
'MiniBatchSize', 1, ...
'InitialLearnRate', 1e-3, ...
Find more on Image Data Workflows in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!Start Hunting!