Classify Function on Neural Network moving slow... and fast?

Folks, sorry this description is so long. However, it's also four code snippets with lots of repition in them so hopefully won't take too long to read.

In these first two snippets, where 1080 images are processed, the Neural Network (transfer learning applied onto resnet50() for testing purposes) takes 63 seconds vs. a SVM which takes 109 seconds.

1) Neural Network (test)

load netTransfer
inputSize = netTransfer.Layers(1).InputSize(1:2);
myDir = uigetdir; %gets directory
myFiles = dir(fullfile(myDir,'*.jpg'));
numfiles = length(myFiles);
tic
for i = 1:numfiles %Please excuse looping to add each image instead of datastore, I was having some problem with the datastore and I do not remember right now what it was 
    img = imread(fullfile(myDir,myFiles(i).name));
    img = imresize(img,inputSize(1:2)); 
    imgarray(:,:,:,i)=img;
end
augmentedimg = augmentedImageDatastore(inputSize,imgarray); %I had replaced the above attempt at datastore after this line already existed, so the resizing is redundant, but shouldn't make a difference
prediction = classify(netTransfer,augmentedimg);
toc

2) SVM (test)

load classifier
inputSize = net.Layers(1).InputSize(1:2);
myDir = uigetdir; %gets directory
myFiles = dir(fullfile(myDir,'*.jpg'));
numfiles = length(myFiles);
tic
for i = 1:numfiles %Please excuse looping to add each image instead of datastore, I was having some problem with the datastore and I do not remember right now what it was 
    img = imread(fullfile(myDir,myFiles(i).name));
    img = imresize(img,inputSize(1:2));
    imgarray(:,:,:,i) = img;
end

    augmentedimg = augmentedImageDatastore(inputSize,imgarray,'ColorPreprocessing','gray2rgb'); %I had replaced the above attempt at datastore after this line already existed, so the resizing is redundant, but shouldn't make a difference
    imageFeatures = activations(net, augmentedimg, ExtractLayer, 'MiniBatchSize', 32,'OutputAs', 'columns');
    [label, score] = predict(classifier,imageFeatures,'ObservationsIn','columns');
toc

Not that I think it would make a difference, but note that around a random 50% of the 1080 images where both used by the SVM and the Network during their respective trainings.

Thinking this means that the Neural Network is much faster, I decided to change from SVM (same one as in #2 above) to the Neural Network (same one as #1 above) iin a program where I take a video feed, segment it for objects, and identify the objects. A max of 10 objects at a time are fed to the classification part. Snippets below:

1) SVM (Main Program)

    for i = 1:size(centroids, 1)
        bbox = bboxes(i, :);
        img = imcrop(frame,bbox);
        img = imresize(img,inputSize(1:2));
        imgarray(:,:,:,i) = img;
    end
    if exist('imgarray')
        augmentedimg = augmentedImageDatastore(inputSize,imgarray,'ColorPreprocessing','gray2rgb');
        imageFeatures = activations(net, augmentedimg, ExtractLayer, 'MiniBatchSize', 32,'OutputAs', 'columns');
        label = predict(classifier,imageFeatures,'ObservationsIn','columns');

2) Neural Network (Main Program)

    for i = 1:size(centroids, 1)
        bbox = bboxes(i, :);
        img = imcrop(frame,bbox);
        img = imresize(img,inputSize);
        imgarray(:,:,:,i) = img;
    end
    if exist('imgarray')
        augmentedimg = augmentedImageDatastore(inputSize,imgarray,'ColorPreprocessing','gray2rgb');
        label = classify(netTransfer,augmentedimg);

There is much more after the second for loop in both snippets, but regardless, no differences in the code before or after the snippets and even within what you see in the snippets except for the classifier. Somehow though, each loop (this consists of the classifcation plus everything else) in the SVM implementation of the code runs between 0.5-1 second, vs. the Neural Net implementation at 2-3 seconds.

So.... how come #2 above is so much slower? I figured it might be an overhead thing, especially given the much smaller data set to identify at a time, as opposed to the 1080 earlier. However, I then ran a test code to compare speeds of the neural network vs. the SVM identifying random images one by one, and the network (except for the first image) was atleast 5x faster.

I have no formal training in Computer Vision and have only googled my way upto this point so please excuse any conceptual errors. Thanks in advance!

[Matt J]

The profiler is needed here, to see what's really going on.

Answer 1

[Avadhoot]

Hi,

As you have correctly stated in your question, the problem appears to be related to an overhead. Your experiments suggest that the neural network is faster at processing the data than the SVM, but when smaller dataset is given as an input, the neural network performs worse than the SVM.

This might be caused by many factors, some of which I have mentioned below:

1. Batch processing capabilities: A neural network is optimized for batch processing unlike the SVM. More so if there is a GPU involved. So, it performs better on large datasets, but falters on smaller datasets as the parallel processing capabilities are not used fully. Try accumulating more images in a batch before processing them in a neural network.
2. Overheads: Creating an augmented datastore is an overhead which is manageable if it is done once in the program. But doing it for multiple times has an adverse effect on the efficiency.
3. Initialization overheads: Neural networks sometimes require initializations which can take some time. You could run a dummy classification model before your main loop to ensure that the initialization overheads are not counted in your measurements.
4. MATLAB profiler: And lastly, as Matt suggested, it is suggested to use the profiler to figure out the bottlenecks in the code to optimize it properly.

I hope it helps.