Where and with how much data is the initial normalization done when training a network

1 view (last 30 days)
When I start a network training (say VGG16) there is an initial step of data normalization. I was expecting all the dataset to load on the RAM, perform the corresponding preprocessing (done by the CPU) and later on, whenever it is needed, compose each batch, do the required additional preprocessing over the batch (done again by the CPU) and then transfer it to the GPU memory. However, seeing the behaviour while running my procedure, I can see that the RAM load does not go so high when doing the aforementioned data normalization (zeromean in this case). Why is this? Am I missing something? Is this initial preprocessing done per batch and stored again in the main memory or something like that?
Thank you so much in advance,
(By the way, sorry if this is a so basic question).
  1 Comment
yanqi liu
yanqi liu on 21 Dec 2021
Edited: yanqi liu on 21 Dec 2021
yes,sir,i aslo has this confused, may be the imageDatastore and augmentedImageDatastore use on cpu to prepare data,and then make them to gpu and train

Sign in to comment.

Answers (1)

Sahil Jain
Sahil Jain on 23 Dec 2021
Hi Iker, the mean and standard deviation statistics are calculated iteratively by computing them for each mini-batch and then merging them with the running statistics. Since it's done batch-wise, not all the data is loaded onto the RAM so RAM usage isn't high. These calculated statistics are then used by the imageInputLayer to perform normalization.

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by