Correct weight Initialization in CNN

26 views (last 30 days)
When a very deep DAG network is built from scratch, the initialization of the weights made by matlab is not very good since it presents a vanishing gradient problem which causes the CNN not to learn.
What is the function with which Matlab does the initiation of CNN weights?
Why do you implement initialization functions in Matlab such as XAVIER or RELU AWARE SCALALED?
Thank you for your answers.
Yuze Zou
Yuze Zou on 3 Jul 2019
I guess this issuse has been solved in the latest release (R2019a) via new default weights initialization method (i.e., Xavier/Glorot) for `fullyConnectedLayer`. You can find more details here.

Sign in to comment.

Accepted Answer

Maria Duarte Rosa
Maria Duarte Rosa on 5 Jul 2019
Edited: Maria Duarte Rosa on 5 Jul 2019
In R2019a, the following weight initializers are available (including a custom initializer via a function handle):
'glorot' (default) | 'he' | 'orthogonal' | 'narrow-normal' | 'zeros' | 'ones' | function handle
Glorot is also know as Xavier initializer.
Here is a page comparing 3 initializers when training LSTMs:
I hope this helps,

More Answers (2)

Andres Ramirez
Andres Ramirez on 31 Jul 2018
Hello Gerg, thanks for answering ... I'll explain:
I have built very deep networks such as: Googlenet, Resnet and VGG19, and I want to train them from scratch with my databases; However, when I do the training of any of these networks, the network does not learn and only reaches a maximum performance of 12 or 15%. I think the low performance is mainly due to the random initialization of the weights made by defaul in matlab does not work for very deep networks, according to the literature, random initialization causes a vanishing gradient problem which causes the network do not learn.
For the above, my questions are:
Why do not implement in matlab more appropriate weight initialization functions for the training of deep DAG networks, for example, XAVIER or RELA AWARE SCALALED?
I hope to have been clear the explanation of the problem I have ...
Thank you.
  1 Comment
Greg Heath
Greg Heath on 1 Aug 2018
Edited: Greg Heath on 1 Aug 2018
Do you have a reference for
I have no idea what this is.

Sign in to comment.

fareed jamaluddin
fareed jamaluddin on 4 Aug 2018
I think you can take a look at this example
I am also looking for a way on weight initialization options, you can see in the example it create the initialization with He method for every conv layer.




Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by