R-CNN Deep Learning with 3D Data

Is it possible to "Train Object Detector Using R-CNN Deep Learning" with 3D data?

3D data: [x,y,z] and not [x,y,channel]

In this case, how do we define de "bounding boxes"?

For the 2D case is something like [x y width height]. And for a set of 2D images (i.e., 3D data)?

Any help please??

Thank you very much in advance!!

Best regards

Answer 1

[Shashank Gupta]

Hi Ana,

Generalising the 2d RCNN model to 3d is not so easy as it seems. the relevance that you intent to make from 2d and then trying to create a 3d architecture may not result in good performance, Although you can give a shot. So, there are few things you need to change. The input of 3d data should look something in the format [x,y,z,channel,batch_size] and the bounding boxes here will be cuboid, so the format will look somthing like [XMIN YMIN ZMIN WIDTH HEIGHT DEPTH]. You also need to change the layer to their respective 3d layers and write the custom training loop to train.

Hope this sounds good or atleast I provide you enough information to explore.

Cheers.

[AnaM]

Hello!

Thank you very much for your answer!

So...If I try "3D" faster R-CNN, how can I use the roiMaxPoolingLayer? Because, as far as I know, only a roiMaxPooling2dLayer is available.

THANK YOU VERY MUCH!!!

[Shashank Gupta]

Hi Ana,

Afaik MATLAB doesn't have the 3D version of roi pooling layer. I think you need to create your own version. take help from roiMaxPooling2dLayer source code and try to generalize it to 3D. It shouldn't be that much difficult to execute. Try out once.

Cheers.

[AnaM]

OK, thank you! I will try that!!

[AnaM]

Hello!!

Still in the context of faster r-cnn, I am trying to train the network (2D), but I always get empty detection results...

- I use a pre-trained backbone network on my data (which has an accuracy of around 70%);

- The images have size [512 512 1] and are uint8 (as well as the input size of the network);

- The bounding boxes are approximately between 30x30 to 60x60;

- I have 2 classes of objects;

- 250 epochs (already varied it but the result is the same) with MB size 64;

- I've tried it with a very low positive overlap range ([0.1 1]);

I used fasterRCNNLayers to create a faster R-CNN object.

(500 images+bounding boxes to train)

1) Is there a problem with the images being grayscale and not in color?

2) Do I have to have bounding boxes from a region other than the object? (as described here: https://www.mathworks.com/matlabcentral/answers/500950-bounding-box-not-drawn-some-variables-are-empty).

I apologize for asking this question in this topic!

Thank you so so much in advance!!!