Deep Learning Prediction with NVIDIA TensorRT

This example demonstrates code generation for a deep learning application using NVIDIA TensorRT™ library. It uses the codegen command to generate a MEX file to run prediction with a ResNet-50 image classification network using TensorRT. A second example demonstrates usage of codegen command to generate a MEX file that runs 8-bit integer prediction with TensorRT for a logo classification network.

Prerequisites

  • CUDA® enabled NVIDIA® GPU with compute capability 3.2 or higher. For the 8-bit integer example, NVIDIA® GPU with compute capability 6.1 or higher is required.

  • NVIDIA CUDA toolkit and driver.

  • NVIDIA cuDNN library (v7).

  • NVIDIA TensorRT library.

  • Environment variables for the compilers and libraries. For information on the supported versions of the compilers and libraries, see Third-party Products. For setting up the environment variables, see Environment Variables.

  • GPU Coder Interface for Deep Learning support package for deep learning code generation.

  • Deep Learning Toolbox™ for using DAGNetwork object.

  • ResNet-50 support package of the Deep Learning Toolbox™ for using pre-trained ResNet network.

Verify the GPU Environment

Use the coder.checkGpuInstall function and verify that the compilers and libraries needed for running this example are set up correctly.

envCfg = coder.gpuEnvConfig('host');
envCfg.DeepLibTarget = 'tensorrt';
envCfg.DeepCodegen = 1;
envCfg.Quiet = 1;
coder.checkGpuInstall(envCfg);

Create a Folder and Copy Relevant Files

The following code creates a folder in your current working folder (pwd). The new folder contains only the files that are relevant for this example. If you do not want to affect the current folder (or if you cannot generate files in this folder), change your working folder.

gpucoderdemo_setup('gpucoderdemo_tensorrt_target');

About 'resnet_predict' Function

We use the popular DAG network ResNet-50 to show image classification with TensorRT. A pretrained ResNet-50 model for MATLAB™ is available in the ResNet-50 support package of the Deep Learning Toolbox. To download and install the support package, use the Add-On Explorer. To learn more about finding and installing add-ons, see Get Add-Ons (MATLAB).

The function resnet_predict.m loads the ResNet-50 network into a persistent network object. On subsequent calls to the function, the same persistent network object is reused.

Run MEX Code Generation for 'resnet_predict' Function

To generate code from the resnet_predict function, use a coder.gpuConfig object with a deep learning configuration object created for TensorRT target library.

cfg = coder.gpuConfig('mex');
cfg.TargetLang = 'C++';
cfg.DeepLearningConfig = coder.DeepLearningConfig('tensorrt');
codegen -config cfg resnet_predict -args {ones(224,224,3)} -report
Code generation successful: To view the report, open('codegen/mex/resnet_predict/html/report.mldatx').

Call Predict on a Test Image

im = imread('peppers.png');
im = imresize(im, [224,224]);
predict_scores = resnet_predict_mex(double(im));
%
% get top 5 probability scores and their labels
[val,indx] = sort(predict_scores, 'descend');
scores = val(1:5)*100;
net = resnet50;
classnames = net.Layers(end).ClassNames;
labels = classnames(indx(1:5));

Clear the static network object loaded in memory.

clear mex;

Generate TensorRT Code for INT8 Prediction

We can also generate TensorRT code that runs inference in int8 precision. We use a pretrained logo classification network to classify logos in images.

Download the pretrained LogoNet network and save it as logonet.mat, if it does not exist. The network was developed in MATLAB and its architecture is similar to that of AlexNet. This network can recognize 32 logos under various lighting conditions and camera angles. The network is pretrained in single precision floating-point.

net = getLogoNet();

TensorRT requires a calibration data set to calibrate a network that is trained in floating-point to compute inference 8-bit integer precision. Set the data type to int8 and the path to the calibration data set using DeepLearningConfig object as shown below. ''logos_dataset'' is a subfolder containing images grouped by their corresponding classification labels. For int8 support, GPU compute capability must be 6.1 and above.

cfg = coder.gpuConfig('mex');
cfg.TargetLang = 'C++';
cfg.GpuConfig.ComputeCapability = '6.1';
cfg.DeepLearningConfig = coder.DeepLearningConfig('tensorrt');
cfg.DeepLearningConfig.DataType = 'int8';
cfg.DeepLearningConfig.DataPath = 'logos_dataset';
cfg.DeepLearningConfig.NumCalibrationBatches = 50;
codegen -config cfg logonet_predict -args {ones(227,227,3,'int8')} -report
Code generation successful: To view the report, open('codegen/mex/logonet_predict/html/report.mldatx').

Run INT8 Prediction on a Test Image

im = imread('test.png');
im = imresize(im, [227,227]);
predict_scores = logonet_predict_mex(int8(im));
%
% get top 5 probability scores and their labels
[val,indx] = sort(predict_scores, 'descend');
scores = val(1:5)*100;
classnames = net.Layers(end).ClassNames;
labels = classnames(indx(1:5));

Clear the static network object loaded in memory.

clear mex;