Code Generation for Deep Learning Networks

This example shows how to perform code generation for an image classification application that uses deep learning. It uses the codegen command to generate a MEX function that runs prediction by using image classification networks such as MobileNet-v2, ResNet, and GoogLeNet.

Prerequisites

  • CUDA® enabled NVIDIA® GPU with compute capability 3.2 or higher.

  • NVIDIA CUDA toolkit and driver.

  • NVIDIA cuDNN library.

  • Environment variables for the compilers and libraries. For information on the supported versions of the compilers and libraries, see Third-party Products (GPU Coder). For setting up the environment variables, see Setting Up the Prerequisite Products (GPU Coder).

  • GPU Coder Interface for Deep Learning Libraries support package. To install this support package, use the Add-On Explorer.

Verify GPU Environment

Use the coder.checkGpuInstall function to verify that the compilers and libraries necessary for running this example are set up correctly.

envCfg = coder.gpuEnvConfig('host');
envCfg.DeepLibTarget = 'cudnn';
envCfg.DeepCodegen = 1;
envCfg.Quiet = 1;
coder.checkGpuInstall(envCfg);

mobilenetv2_predict Entry-Point Function

MobileNet-v2 is a convolutional neural network that is trained on more than a million images from the ImageNet database. The network is 155 layers deep and can classify images into 1000 object categories, such as keyboard, mouse, pencil, and many animals. The network has an image input size of 224-by-224. Use the analyzeNetwork function to display an interactive visualization of the deep learning network architecture.

net = mobilenetv2();
analyzeNetwork(net);

The mobilenetv2_predict.m entry-point function takes an image input and runs prediction on the image using the pretrained MobileNet-v2 convolutional neural network. The function uses a persistent object mynet to load the series network object and reuses the presistent object for prediction on subsequent calls.

type('mobilenetv2_predict.m')
% Copyright 2017-2019 The MathWorks, Inc.

function out = mobilenetv2_predict(in) 
%#codegen

persistent mynet;

if isempty(mynet)
    mynet = coder.loadDeepLearningNetwork('mobilenetv2','mobilenetv2');
end

% pass in input   
out = mynet.predict(in);

Run MEX Code Generation

To generate CUDA code for the mobilenetv2_predict entry-point function, create a GPU code configuration object for a MEX target and set the target language to C++. Use the coder.DeepLearningConfig function to create a CuDNN deep learning configuration object and assign it to the DeepLearningConfig property of the GPU code configuration object. Run the codegen command and specify an input size of [224,224,3]. This value corresponds to the input layer size of the MobileNet-v2 network.

cfg = coder.gpuConfig('mex');
cfg.TargetLang = 'C++';
cfg.DeepLearningConfig = coder.DeepLearningConfig('cudnn');
codegen -config cfg mobilenetv2_predict -args {ones(224,224,3)} -report
Code generation successful: To view the report, open('codegen/mex/mobilenetv2_predict/html/report.mldatx').

Generated Code Description

The series network is generated as a C++ class containing an array of 155 layer classes and functions to set up, call predict, and clean up the network.

class b_mobilenetv2_0
{
   ....
   public:
     b_mobilenetv2_0();
     void setup();
     void predict();
     void cleanup();
     ~b_mobilenetv2_0();
};

The setup() method of the class sets up handles and allocates memory for each layer of the network object. The predict() method performs prediction for each of the 155 layers in the network.

The entry-point function mobilenetv2_predict() in the generated code file mobilenetv2_predict.cu constructs a static object of b_mobilenetv2 class type and invokes setup and predict on this network object.

static b_mobilenetv2_0 mynet;
static boolean_T mynet_not_empty;
/* Function Definitions */
void mobilenetv2_predict(const real_T in[150528], real32_T out[1000])
{
  if (!mynet_not_empty) {
    DeepLearningNetwork_setup(&mynet);
    mynet_not_empty = true;
  }
   /*  pass in input    */
   DeepLearningNetwork_predict(&mynet, in, out);
 }

Binary files are exported for layers with parameters such as fully connected and convolution layers in the network. For instance, files cnn_mobilenetv2_conv*_w and cnn_mobilenetv2_conv*_b correspond to weights and bias parameters for the convolution layers in the network. To see a list of the generated files, use:

dir(fullfile(pwd, 'codegen', 'mex', 'mobilenetv2_predict'))

Run Generated MEX

Load an input image.

im = imread('peppers.png');
imshow(im);

Call mobilenetv2_predict_mex on the input image.

im = imresize(im, [224,224]);
predict_scores = mobilenetv2_predict_mex(double(im));

Get the top five prediction scores and their labels.

[scores,indx] = sort(predict_scores, 'descend');
classNames = net.Layers(end).ClassNames;
classNamesTop = classNames(indx(1:5));

h = figure;
h.Position(3) = 2*h.Position(3);
ax1 = subplot(1,2,1);
ax2 = subplot(1,2,2);

image(ax1,im);
barh(ax2,scores(5:-1:1))
xlabel(ax2,'Probability')
yticklabels(ax2,classNamesTop(5:-1:1))
ax2.YAxisLocation = 'right';
sgtitle('Top Five Predictions That Use MobileNet-v2')

Classification of Video

The included helper function mobilenet_live.m grabs frames from a webcam, performs prediction, and displays the classification results on each of the captured video frames. This example uses the function that is supported by the MATLAB® Support Package for USB Webcams™. You can download and install the support package through the Support Package Installer.

type('mobilenet_live.m')
% Copyright 2017-2019 The MathWorks, Inc.

function mobilenet_live

% Connect to a camera
camera = webcam; 

% The labels with top 5 prediction scores are 
% mapped to corresponding labels
net = mobilenetv2();
classnames = net.Layers(end).ClassNames;

imfull = zeros(224,400,3, 'uint8');

fps = 0;

ax = axes;

while true
    % Take a picture
    ipicture = camera.snapshot;       
    
    % Resize and cast the picture to single        
    picture = imresize(ipicture,[224,224]);  
    
    % Call MEX function for MobileNet-v2 prediction
    tic;    
    pout = mobilenetv2_predict(single(picture));
    newt = toc;
       
    % fps 
    fps = .9*fps + .1*(1/newt);
    
    % top 5 scores
    [top5labels, scores] = getTopFive(pout,classnames);
    
    % display
    if isvalid(ax)
        dispResults(ax, imfull, picture, top5labels, scores, fps);   
    else
        break;
    end
end

end

function dispResults(ax, imfull, picture, top5labels, scores, fps)
for k = 1:3
    imfull(:,177:end,k) = picture(:,:,k);
end

h = imshow(imfull, 'InitialMagnification',200, 'Parent', ax);
scol = 1;
srow = 20;
text(get(h, 'Parent'), scol, srow, sprintf('MobileNet-v2 Demo'), 'color', 'w', 'FontSize', 20);
srow = srow + 20;

text(get(h, 'Parent'), scol, srow, sprintf('Fps = %2.2f', fps), 'color', 'w', 'FontSize', 15);
srow = srow + 20;
for k = 1:5
    t = text(get(h, 'Parent'), scol, srow, top5labels{k}, 'color', 'w','FontSize', 15);
    pos = get(t, 'Extent');
    text(get(h, 'Parent'), pos(1)+pos(3)+5, srow, sprintf('%2.2f%%', scores(k)), 'color', 'w', 'FontSize', 15);
    srow = srow + 20;
end

drawnow;
end

function [labels, scores] = getTopFive(predictOut,classnames)
[val,indx] = sort(predictOut, 'descend');
scores = val(1:5)*100;
labels = classnames(indx(1:5));
end

Clear the static network object that was loaded in memory.

clear mex;

Classification of Images by Using ResNet-50 network

You can also use the DAG network ResNet-50 for image classification. A pretrained ResNet-50 model for MATLAB is available in the ResNet-50 support package of Deep Learning Toolbox. To download and install the support package, use the Add-On Explorer. To learn more about finding and installing add-ons, see Get Add-Ons (MATLAB).

net = resnet50;
disp(net)
  DAGNetwork with properties:

         Layers: [177×1 nnet.cnn.layer.Layer]
    Connections: [192×2 table]
     InputNames: {'input_1'}
    OutputNames: {'ClassificationLayer_fc1000'}

Run MEX Code Generation

To generate CUDA code for the resnet_predict.m entry-point function,create a GPU code configuration object for a MEX target and set the target language to C++. This entry-point function calls the resnet50 function to load the network and perform prediction on the input image.

cfg = coder.gpuConfig('mex');
cfg.TargetLang = 'C++';
cfg.DeepLearningConfig = coder.DeepLearningConfig('cudnn');
codegen -config cfg resnet_predict -args {ones(224,224,3)} -report
Code generation successful: To view the report, open('codegen/mex/resnet_predict/html/report.mldatx').

Call resnet_predict_mex on the input image.

predict_scores = resnet_predict_mex(double(im));

Get the top five prediction scores and their labels.

[scores,indx] = sort(predict_scores, 'descend');
classNames = net.Layers(end).ClassNames;
classNamesTop = classNames(indx(1:5));

h = figure;
h.Position(3) = 2*h.Position(3);
ax1 = subplot(1,2,1);
ax2 = subplot(1,2,2);

image(ax1,im);
barh(ax2,scores(5:-1:1))
xlabel(ax2,'Probability')
yticklabels(ax2,classNamesTop(5:-1:1))
ax2.YAxisLocation = 'right';
sgtitle('Top Five Predictions That Use ResNet-50')

Clear the static network object that was loaded in memory.

clear mex;

Classification of Images by Using GoogLeNet (Inception) network

A pretrained GoogLeNet model for MATLAB is available in the GoogLeNet support package of Deep Learning Toolbox. To download and install the support package, use the Add-On Explorer. To learn more about finding and installing add-ons, see Get Add-Ons (MATLAB).

net = googlenet;
disp(net)
  DAGNetwork with properties:

         Layers: [144×1 nnet.cnn.layer.Layer]
    Connections: [170×2 table]
     InputNames: {'data'}
    OutputNames: {'output'}

Run MEX Code Generation

Generate CUDA code for the googlenet_predict.m entry-point function. This entry-point function calls the googlenet function to load the network and perform prediction on the input image. To generate code for this entry-point function, create a GPU configuration object for MEX target.

cfg = coder.gpuConfig('mex');
cfg.TargetLang = 'C++';
cfg.DeepLearningConfig = coder.DeepLearningConfig('cudnn');
codegen -config cfg googlenet_predict -args {ones(224,224,3)} -report
Code generation successful: To view the report, open('codegen/mex/googlenet_predict/html/report.mldatx').

Call googlenet_predict_mex on the input image.

im = imresize(im, [224,224]);
predict_scores = googlenet_predict_mex(double(im));

Get the top five prediction scores and their labels.

[scores,indx] = sort(predict_scores, 'descend');
classNames = net.Layers(end).ClassNames;
classNamesTop = classNames(indx(1:5));

h = figure;
h.Position(3) = 2*h.Position(3);
ax1 = subplot(1,2,1);
ax2 = subplot(1,2,2);

image(ax1,im);
barh(ax2,scores(5:-1:1))
xlabel(ax2,'Probability')
yticklabels(ax2,classNamesTop(5:-1:1))
ax2.YAxisLocation = 'right';
sgtitle('Top Five Predictions That Use GoogLeNet')

Clear the static network object that was loaded in memory.

clear mex;

Related Topics