How to add a regression layer at the end of an existing CNN?

Hi

I have been trying to use transfer learning with a regression layer at the end. However, I end up with NaNs when I try to predict new images with the ConvNet. How should I put together the layers at the end? Like this?

% Load VGG network
net = vgg19; 

% Remove last 3 layers.
layersTransfer = net.Layers(1:end-3);

% Add a fully connected layer and a regression layer
layers = [layersTransfer
          fullyConnectedLayer(1)
          regressionLayer];

% Training options
transferLayerOptions = trainingOptions('sgdm','MiniBatchSize',25,...
                                       'InitialLearnRate',0.001, ...
                                       'MaxEpochs',7, 'Verbose', true);

[Birju Patel]

What happened during training? Did you see NaNs printed in the training log printed in the command window? If so, it's likely your learning rate is too high.

[Tobias Pahlberg]

Yes! I tried lowering the initial learning rate first to 1e-4 but had the same problem, then I lowered to 1e-5 and I got numbers now. Could you explain why I need such a low rate here, is it because of the size of the network? Here is what the training log looks like now.

|=========================================================================================|
|     Epoch    |   Iteration  | Time Elapsed |  Mini-batch  |  Mini-batch  | Base Learning|
|              |              |  (seconds)   |     Loss     |     RMSE     |     Rate     |
|=========================================================================================|
|            1 |            1 |         7.98 |     366.5998 |        27.08 |     1.00e-05 |
|            1 |           50 |     30791.43 |     338.3933 |        26.02 |     1.00e-05 |
|            1 |          100 |     31443.52 |      73.1415 |        12.09 |     1.00e-05 |
|            1 |          150 |     32114.34 |      74.8708 |        12.24 |     1.00e-05 |
|            1 |          200 |     32688.59 |      26.7964 |         7.32 |     1.00e-05 |
|            2 |          250 |     33330.43 |      28.9653 |         7.61 |     1.00e-05 |
|            2 |          300 |     33941.49 |      29.8938 |         7.73 |     1.00e-05 |
|            2 |          350 |     34541.52 |      27.4190 |         7.41 |     1.00e-05 |
|            2 |          400 |     35179.22 |      28.7636 |         7.58 |     1.00e-05 |
|            3 |          450 |     35758.08 |      32.7189 |         8.09 |     1.00e-05 |
|            3 |          500 |     36335.59 |      18.0422 |         6.01 |     1.00e-05 |
|            3 |          550 |     37002.46 |      26.7821 |         7.32 |     1.00e-05 |
|            3 |          600 |     37619.53 |      17.1531 |         5.86 |     1.00e-05 |
|            4 |          650 |     38210.78 |      17.5978 |         5.93 |     1.00e-05 |
|            4 |          700 |     38796.26 |      13.0254 |         5.10 |     1.00e-05 |
|            4 |          750 |     39360.78 |      15.2743 |         5.53 |     1.00e-05 |
|            4 |          800 |     39947.61 |      21.7468 |         6.59 |     1.00e-05 |
|            5 |          850 |     40587.68 |      17.7648 |         5.96 |     1.00e-05 |
|            5 |          900 |     41240.12 |       9.3954 |         4.33 |     1.00e-05 |
|            5 |          950 |     41841.30 |      23.7572 |         6.89 |     1.00e-05 |
|            5 |         1000 |     42466.40 |       9.1774 |         4.28 |     1.00e-05 |
|            6 |         1050 |     43082.04 |      10.6128 |         4.61 |     1.00e-05 |
|            6 |         1100 |     43654.45 |       8.0469 |         4.01 |     1.00e-05 |
|            6 |         1150 |     44324.75 |      13.1615 |         5.13 |     1.00e-05 |
|            6 |         1200 |     44899.66 |       4.5212 |         3.01 |     1.00e-05 |
|            7 |         1250 |     45491.29 |      13.4922 |         5.19 |     1.00e-05 |
|            7 |         1300 |     46126.86 |       5.1565 |         3.21 |     1.00e-05 |
|            7 |         1350 |     46704.71 |      14.9740 |         5.47 |     1.00e-05 |
|            7 |         1400 |     47259.80 |       6.1847 |         3.52 |     1.00e-05 |
|=========================================================================================|

Answer 1

[Tobias Pahlberg]

As Birju Patel suggested. I believe the problem was a too high initial learn rate. Lowering it to 1e-5 worked.