Custom deep learning loop take more memory than using trainNetwork()?

Hi,

I followed the instructions from the link below to create a custom training loop by using a U-Net architecture.

https://www.mathworks.com/help//deeplearning/ug/train-network-using-custom-training-loop.html

By the same network architecture and with same "multi-gpu" setting (I have 2 RTX 2060 GPU), I found that I can only take 4 minibatch size at best in the custom training loop, while 16 minibarch size at best by using the built-in trainNetwork() function.

Is this a normal phenomenon that custom loop training will take more gpu memory than trainNetwork()?

Thanks!

[Sara Ahmed]

Same here :(

Answer 1

[Shashank Gupta]

Yes, it is an expected behaviour, the custom loop does take some extra amount of memory while the existing function trainNetwork is very optimised. More custom loop more inefficiency and thus more GPU memory usage. Neverthless, you can optimise the custom training loop but even then we can't be fully sure that it is as much optimised as trainNetwork.

I hope this clear some of your confusion.

[Qiao Hu]

Hi, Shashank,

Thank you very much for your response.

I have a few more question regarding to the custom loop.

1) For custom loop, I did not see 'multi-GPU' options referring to the reference below. Given the condition of two GPUs in my case, shall I creating custom loop for parallel computing in order to utilize both GPUs?

https://www.mathworks.com/help/deeplearning/ug/specify-training-options-in-custom-training-loop.html

2) Is there any reference available that I can refer to optimize the custom network?

Thanks a bunch!

[Shashank Gupta]

Hey Qiao,

Have a look at this Link, this might enable you to use parallel capabilities in the custom training loop.

Currently, there is no specific reference that talks about the optimisation of custom loop specifically because it is hard to generalise anything and come up with a documented reference. Generally these jobs are really subjective, depends on what sort of things you want to implement, Nevertheless, some suggestions, look for dlarray capable function for quick computing, Try using more MATLAB function rather than implementing your own. try to use as less code as necessary.

[Qiao Hu]

Thanks a bunch!