Why is my neural network performing worse as the number of hidden layers increases?

Hello, I am currently using the Matlab Neural Network toolbox to experiment with the Iris dataset. I am training with "trainlm" algorithm, and I decided to see what would happen if I trained with 1:20 hidden layers. I was not expecting any change in the classification error, but when I do this, I get the following output:

I have been looking for a solution, but I cannot explain why the classification error begins to jump, or even increases at all as the number of hidden layers increases.

Thank You

Answer 1

[Greg Heath]

The ultimate goal is too obtain a net that performs well on non-training data that comes from the same or similar source as the training data. This is called GENERALIZATION.

Frequent causes of failure are

 1. Not enough weights to adequately characterize the training data
 2. Training data does not adequately characterize the salient features of non-training data because of measurement error, transcription error, noise, interference, insufficient sampling size and variability
 3. Fewer training equations than unknown weights.
 4. Random weight initialization

Various techniques use to mitigate these causes are

 1. Remove bad data and outliers (plots help)
 2. Use enough training data to sufficiently characterize non-training data.
 3. Use enough weights to adequately characterize the training data
 4. Use more training equations than unknown weights. The stability of 
solutions w.r.t. noise and errors increases as the ratio increases.
 5. Use the best of multiple random initialization & data-division designs 
 6. K-fold Cross-validation
 7. Validation Stopping
 8. Regularization

For the iris_dataset

 [ I N ] = size(input)  % [ 4 150 ]
 [ O N ] = size(target) % [ 3 150 ]

 Assuming the default 0.7/0.15/0.15 trn/val/tst  data division, the number of training equations is approximately

 Ntrneq = 0.7*N*O   % 315

 Assuming the default I-H-O node topology, the number of unknown equations is

 Nw = (I+1)*H+(H+1)*O = (I+O+1)*H + O

 Obviously, Nw < Ntrneq when H <= Hub  (upper bound) where

 Hub = floor( (Ntrneq-O)/(I+O+1))  % 39

 Expecting decent solutions for H <= 20 seems reasonable. However, to 
mitigate the random initial weights and data division, design 10 nets for each value

 I have posted zillions of examples in both the NEWSGROUP and ANSWERS. I use patternnet for classification.

Hope this helps.

Thank you for formally accepting my answer

Greg

[Greg Heath]

I reread your post after posting my answer.

Since only ONE hidden layer is necessary and sufficient, your statement of 1:20 hidden LAYERS was misread by me as hidden NODES on ONE hidden layer.

Did I interpret your intention correctly?

Greg

[Greg Heath]

A QUICKIE:

 close all, clear all, clc
 [ x, t  ] = iris_dataset;
 MSE00 = mean(var(t',1))%0.22222
 rng(4151941)
 for h = 1:24   
    net = patternnet(h);
    [ net tr y e ]= train(net,x,t);
    R2(1,h) = 1-mse(e)/MSE00;
 end
 R2 = R2
% R2 = 
% 0.528  0.952 0.924 0.925  0.955  0.955
% 0.953  0.921 0.961 0.956  0.928  0.883
% 0.959  0.943 0.954 0.951  0.961  0.946
% 0.968  0.951 0.940 0.952  0.951  0.913

Hope this helps.

Greg

[bitslice]

Hello Greg, and thank you very much for the insightful response.

Although one hidden layer was sufficient, I decided to test the toolbox by extending to additional hidden layers. I was surprised to see that the error actually increased when I did this as I was under the impression that additional hidden layers would only decrease the error, especially on the training set. This is where the confusion started for me. Although I understand the concept of generalization to new data, I am still not entirely sure why my code consistently has high error at those specific hidden layer depths.

All of this information is helping me get on the right track, but I am still a bit confused regarding this error.

[bitslice]

Also, I used only one node in each of these hidden layers.

[bitslice]

Ok, with your tips I was able to figure this out:

Changing the random seed does indeed change the consistency of the results, implying that the increased error is due to the random weight initialization.

Thank you so much!

Answer 2

[Walter Roberson]

Each layer is initialized randomly. If you do not provide enough data to train the effects of the randomness out, then you have the effect of the cumulative randomness.

[bitslice]

Thank you so much for the quick response. Do you know where I can find more information on this? At this point, I am not sure how I would go about calculating the amount of training data needed to remove the randomness.

[Walter Roberson]

Greg Heath has written several times about the amount of data that one should use, but I cannot think of good keywords at the moment to search for.

[Greg Heath]

 Try combinations of Ntrneq Nw Hub