Why does Network Analyzer warn me network architecture is not supported by trainNetwork when I follow the exact steps of the fasterRCNNLayers example?

I'm trying to train a Faster R-CNN object detector using the ResNet-50 pretrained network. Upon initiating the training, I've been getting the following error:

Error using visiongpuROIMaxPoolingForward

Maximum variable size allowed on the device is exceeded.

At first I thought it might have something to do with the fact that my training data is being sampled from multiple large blocked images (23 images of dimensions [9504 x 6336 x 3] and type uint8, to be exact) in a blockedImageDatastore, using a blockLocationSet with the BlockSize set to the network's inputImageSize, in this case [224,224]. I thought maybe what was happening was that all 23 entire images needed to be simultaneously loaded into the GPU's memory in order to extract the blocks specified in the blockLocationSet and then pass them along to the trainFasterRCNNObjectDetector function, which may be exceeding the hard limit of intmax('int32') = 2,147,483,647 elements in a GPU array, given that [9504 x 6336 x 3] x 23 = 4,154,996,736.

But even after reducing the number of blocked training images to 5 ([9504 x 6336 x 3] x 5 = 903,260,160), I was still getting the error. And even after reducing the BlockSize/inputImageSize as well as the MiniBatchSize to conservatively small numbers that I've definitely pulled off before when doing other deep learning tasks such as scene classification and semantic segmentation, I was still getting the error. My GPU is a GeForce RTX 3060 with 12 GB of memory, which is a pretty decent amount.

Then I started looking at the Network Analyzer, and noticed the warning:

"This network architecture is not supported by trainNetwork."

I wasn't sure why, so as a basic test I tried following the exact example given on the fasterRCNNLayers help page. So literally exactly as in the example:

inputImageSize = [224 224 3];
numClasses = 1;
network = 'resnet50';
featureLayer = 'activation_40_relu';
anchorBoxes = [64,64; 128,128; 192,192];
lgraph = fasterRCNNLayers(inputImageSize,numClasses,anchorBoxes, ...
                          network,featureLayer)

And then:

analyzeNetwork(lgraph)

And I still get the "network architecture is not supported by trainNetwork" warning! What the?!?

With a bit more digging, I noticed that the ROIMaxPoolingLayer argument is possibly not working the way it's supposed to. Whether I leave it to the default 'auto' (by not specifying the argument at all) or manually set it to 'insert', upon inspection of the network in Network Analyzer, it doesn't seem to insert an ROI max pooling layer after the specified feature extraction layer ('activation_40_relu') in the same way as shown in Faster R-CNN Examples under Add ROI Max Pooling Layer, which also uses ResNet-50. In this example, the network is supposed to look like this:

And then like this after adding the ROI input layer:

Instead, the network I created using the code from the fasterRCNNLayers example looks like this:

And given that "ROIMaxPooling" is part of the error I've been getting when I try to initiate network training, I'm now wondering if this might be in some way responsible and what I might be able to do to fix it?

[Joss Knight]

Usually if you can't get a doc example to work it is because you have an older version of MATLAB. Try visiting the archive documentation for your MATLAB version and getting the code from there, or alternatively retrieving the example from your installation using openExample.

Answer 1

[Ranjeet]

Hi Dominique,

The warning message shows that the ‘Faster R-CNN’ network is not supported to be trained using trainNetwork. However, the following examples can be used to train a Faster R-CNN network –

Train a Faster R-CNN Object Detector

Hope this helps!