CUDN_STATU​S_BAD_PARA​M:While running CNN in Matlab

I am currently trying to train and test a network in matlab using cnn using gpu. I have gpu -GTX 1070 and CUDA 8.0 and cudnn 5.1. I followed the example of object detection using deep learning provided by mathworks from the link https://www.mathworks.com/help/vision/examples/object-detection-using-deep-learning.html

After I run the following lines:

        % Train a network.
        Net1 = trainNetwork(trainingImages, trainingLabels, layers, opts);

It is giving me the trained network as output. But after I run the following line

        YTest = predict(Net1, X_test);

it is giving me following error

    Error using nnet.internal.cnngpu.convolveForward2D
Unexpected error calling cuDNN: CUDNN_STATUS_BAD_PARAM.
Error in nnet.internal.cnn.layer.Convolution2D/doForward (line 218)
            Z = nnet.internal.cnngpu.convolveForward2D( ...
Error in nnet.internal.cnn.layer.Convolution2D/forwardNormal (line 195)
            Z = this.doForward(X,this.Weights.Value,this.Bias.Value);
Error in SeriesNetwork/predict (line 139)
                Y(:,:,:,i) = predictNetwork.predict(X);

Any help will be appreciated. Thank you

[Walter Roberson]

Did you install the CUDNN fix for Pascal architecture? https://www.mathworks.com/support/bugreports/1439741

[Akash Maity]

Yes I did but it didn't help.

[Ellen Apolinar]

Did you run the original matlab script to see if it works properly? Because if it works without errors maybe there is something wrong with your X_test and Matlab/GPU/CUDNN are ok.

How does X_test look like?

Btw. always consider the names of your variables (YTest, X_test).

[Patrick Marmaroli]

Hello, while it worked fine up to now, I just experienced the same error. Impossible to make it work again. I don't know how to figure it out. Any help is welcolmed.

Training on single GPU.
Initializing input data normalization.
|========================================================================================|
|  Epoch  |  Iteration  |  Time Elapsed  |  Mini-batch  |  Mini-batch  |  Base Learning  |
|         |             |   (hh:mm:ss)   |   Accuracy   |     Loss     |      Rate       |
|========================================================================================|
Error using trainNetwork (line 165)
Unexpected error calling cuDNN: CUDNN_STATUS_BAD_PARAM.

Here are my GPU properties

gpuDevice

ans = 

  CUDADevice with properties:

                      Name: 'Quadro M620'
                     Index: 1
         ComputeCapability: '5.0'
            SupportsDouble: 1
             DriverVersion: 10.1000
            ToolkitVersion: 10
        MaxThreadsPerBlock: 1024
          MaxShmemPerBlock: 49152
        MaxThreadBlockSize: [1024 1024 64]
               MaxGridSize: [2.1475e+09 65535 65535]
                 SIMDWidth: 32
               TotalMemory: 2.1475e+09
           AvailableMemory: 1.5266e+09
       MultiprocessorCount: 4
              ClockRateKHz: 1017500
               ComputeMode: 'Default'
      GPUOverlapsTransfers: 1
    KernelExecutionTimeout: 1
          CanMapHostMemory: 1
           DeviceSupported: 1
            DeviceSelected: 1
            
            

From the training data side, everything seems correct....

K>> size(XTrain)

ans =

    13     7     1   304

K>> size(YTrain)

ans =

   304     1

K>> sum(sum(sum(isnan(XTrain))))

ans =

     0

K>> sum(isnan(double(YTrain)))

ans =

     0

Thank you for your suggestions.

[Patrick Marmaroli]

Just a quick update about my last comment, in my case the error came from a bad design of the max pool layer.

As shown below my mistake was to apply a 3x3 max pool onto an input image of 4x2.

Unfortunately this error hasn't been detected by the DLN Analyzer (analyzeNetwork).