Code Generation for LSTM Network on Raspberry Pi

This example shows how to generate code that does not depend on any deep learning libraries for a pretrained long short-term memory (LSTM) network. After code generation, you can deploy the code on a Raspberry Pi™ target. In this example, the LSTM network predicts the Remaining Useful Life (RUL) of a machine. The network takes as input time series data sets that represent various sensors in the engine. The network returns the Remaining Useful Life of an engine, measured in cycles, as its output. For more information about the pretrained LSTM network used in this example, see the example Sequence Classification Using Deep Learning (Deep Learning Toolbox).

This example uses the PIL based workflow to generate a MEX function, which in turn calls the executable generated in the target hardware from MATLAB. This example demonstrates two different approaches for performing prediction by using an LSTM network:

• The first approach uses a standard LSTM network and runs inference on a set of time series data.

• The second approach leverages the stateful behavior of the same LSTM network. In this method, you pass a single timestep of data at a time, and have the network update its state at each time step.

Third-Party Prerequisites

• Raspberry Pi hardware

This example is not supported in MATLAB Online.

Download and Prepare Test Data

This section summarizes the steps to download and prepare the test data that this example uses. This example uses the Turbofan Engine Degradation Simulation Data Set as described in [1]. This data set contains 100 training observations and 100 test observations. The training data contains simulated time series data for 100 engines. Each sequence has 17 features, varies in length, and corresponds to a full run to failure (RTF) instance. The test data contains 100 partial sequences and corresponding values of the Remaining Useful Life at the end of each sequence. For more information on the Turbofan Engine Degradation Simulation data set and the preprocessing steps, see the example Sequence-to-Sequence Regression Using Deep Learning (Deep Learning Toolbox).

Create a directory to store the Turbofan Engine Degradation Simulation data set.

dataFolder = fullfile(tempdir,"turbofan");
if ~exist(dataFolder,'dir')
    mkdir(dataFolder);
end

Download and extract the Turbofan Engine Degradation Simulation data set.

filename = matlab.internal.examples.downloadSupportFile("nnet","data/TurbofanEngineDegradationSimulationData.zip");
unzip(filename,dataFolder)

Calculate Mean and Standard Deviation of Training Data

In the following step, you normalize the test predictors using the mean and standard deviation of the training data. So, you must first use the training data to calculate these normalization parameters.

Load the training data, each column is one observation, each row is one feature. Remove the features that have constant values.

filenamePredictors = fullfile(dataFolder,"train_FD001.txt");
[XTrain] = processTurboFanDataTrain(filenamePredictors);

m = min([XTrain{:}],[],2);
M = max([XTrain{:}],[],2);
idxConstant = M == m;

for i = 1:numel(XTrain)
    XTrain{i}(idxConstant,:) = [];
end

Calculate the mean and standard deviation over all observations.

mu = mean([XTrain{:}],2);
sig = std([XTrain{:}],0,2);

Prepare Test Data

Prepare the test data using the function processTurboFanDataTest attached to this example. The function processTurboFanDataTest extracts the data from filenamePredictors and filenameResponses and returns the cell arrays XValidate and YValidate, which contain the test predictor and response sequences, respectively.

filenamePredictors = fullfile(dataFolder,"test_FD001.txt");
filenameResponses = fullfile(dataFolder,"RUL_FD001.txt");
[XValidate,YValidate] = processTurboFanDataTest(filenamePredictors,filenameResponses);

Remove features with constant values using idxConstant calculated from the training data. Normalize the test predictors using the parameters mu and sig calculated from the training data. Clip the test responses at the threshold 150. This same clipping threshold was used on the training data while training the network.

thr = 150;
for i = 1:numel(XValidate)
    XValidate{i}(idxConstant,:) = [];
    XValidate{i} = (XValidate{i} -  mu) ./ sig;
    YValidate{i}(YValidate{i} > thr) = thr;
end

Cast the data to single for deployment.

XValidate = cellfun(@single, XValidate,  'UniformOutput',  false);
YValidate = cellfun(@single, YValidate,  'UniformOutput',  false);

Set Up a Code Generation Configuration Object for a Static Library

To generate a PIL MEX function for a specified entry-point function, create a code configuration object for a static library and set the verification mode to 'PIL'. Set the target language to C.

cfg = coder.config('lib', 'ecoder', true);
cfg.VerificationMode = 'PIL';
cfg.TargetLang = 'C';
cfg.CodeReplacementLibrary = 'GCC ARM Cortex-A';

Set Up a Configuration Object for Deep Learning Code Generation

Create a deep learning configuration object and set target library to none. Attach the deep learning configuration object to the code generation configuration object.

dlcfg = coder.DeepLearningConfig(TargetLibrary = 'none');
cfg.DeepLearningConfig = dlcfg;

Create a Connection to the Raspberry Pi

Use the MATLAB Support Package for Raspberry Pi Support Package function, raspi, to create a connection to the Raspberry Pi. In the following code, replace:

• raspiname with the name of your Raspberry Pi

• username with your user name

• password with your password

r = raspi;

Configure Code Generation Hardware Parameters for Raspberry Pi

Create a coder.Hardware object for Raspberry Pi and attach it to the code generation configuration object.

hw = coder.hardware('Raspberry Pi');
cfg.Hardware = hw;

Generate PIL MEX Function for LSTM Network

In this approach, you generate code for the entry-point function rulPredict.

The rulPredict entry-point function takes an entire time series data set as an input and passes it to the network for prediction. Specifically, the function uses the LSTM network that is trained in the example Sequence Classification Using Deep Learning (Deep Learning Toolbox). The function loads the network object from the rulDlnetwork.mat file into a persistent variable and reuses this persistent object in subsequent prediction calls. A sequence-to-sequence LSTM network enables you to make different predictions for each individual time step of a data sequence. A dlarray object is created within the entry-point function, input and output to the function are of primitive datatypes. For more information, see Code Generation for dlarray.

type('rulPredict.m')

function out = rulPredict(in, dataFormat)
%#codegen

% Copyright 2020-2024 The MathWorks, Inc.

persistent mynet;

if isempty(mynet)
    mynet = coder.loadDeepLearningNetwork('rulDlnetwork.mat');
end

% Construct formatted dlarray input
dlIn = dlarray(in, dataFormat);

dlOut = predict(mynet, dlIn);

out = extractdata(dlOut);

end

The entry-point function rulPredict requires a data format to construct a dlarray input to the dlnetwork. The first dimension of the dlarray, corresponding to the features, is set to channel 'C' and the second one is set to sequence length 'T'.

dataFormat = 'CT';

To generate code by using the codegen command, use the coder.typeof function to specify the type and size of the input argument to the entry-point function. In this example, the input is of single data type with a feature dimension value of 17 and a variable sequence length. Specify the sequence length as variable-size to perform prediction on an input sequence of any length.

matrixInput = coder.typeof(single(0),[17 Inf],[false true]);

Run the codegen command to generate a PIL based mex function rulPredict_pil on the host platform.

codegen -config cfg rulPredict -args  {matrixInput, coder.Constant(dataFormat)} -report

### Connectivity configuration for function 'rulPredict': 'Raspberry Pi'
Location of the generated elf : /home/pi/MATLAB_ws/R2024b/home/yangzhu/Documents/MATLAB/ExampleManager/yangzhu.24bGR2/deeplearning_shared-ex26065676/codegen/lib/rulPredict/pil
Code generation successful: View report

Run Generated PIL MEX Function on Test Data

The XValidate variable contains 100 input observations. Each observation has 17 features with varying sequence length.

XValidate(1:5)

ans=5×1 cell array
    {17×31  single}
    {17×49  single}
    {17×126 single}
    {17×106 single}
    {17×98  single}

The YValidate variable contains 100 output observations that correspond to the XValidate input variable. Each output observation is a Remaining Useful Life (RUI) value, measured in cycles, for each time step data in entire sequence.

YValidate(1:5)

ans=5×1 cell array
    {[                                                                                                                                                                                                                                                                                                                                                             142 141 140 139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112]}
    {[                                                                                                                                                                                                                                                                                       146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98]}
    {[150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69]}
    {[                                                                   150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 88 87 86 85 84 83 82]}
    {[                                                                                          150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 150 149 148 147 146 145 144 143 142 141 140 139 138 137 136 135 134 133 132 131 130 129 128 127 126 125 124 123 122 121 120 119 118 117 116 115 114 113 112 111 110 109 108 107 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91]}

Run the generated MEX function rul_lstmnet_predict_pil on a random test data set.

idx = randperm(numel(XValidate), 1);
inputData = XValidate{idx};

YPred1 = rulPredict_pil(inputData,dataFormat);

### Starting application: 'codegen/lib/rulPredict/pil/rulPredict.elf'
    To terminate execution: clear rulPredict_pil
### Launching application rulPredict.elf...

Compare Predictions with Test Data

Use a plot to compare the MEX output data with the test data.

figure('Name', 'Standard LSTM', 'NumberTitle', 'off');

plot(YValidate{idx},'--')
hold on
plot(YPred1,'.-')
hold off

ylim([0 175])
title("Test Observation " + idx)
xlabel("Time Step")
ylabel("RUL measured in cycles")

Clear PIL

Terminate the PIL execution process.

clear rulPredict_pil;

### Host application produced the following standard output (stdout) and standard error (stderr) messages:

Generate PIL MEX Function for Stateful LSTM Network

Instead of passing the entire timeseries to predict in one step, you can run prediction on an input by streaming in one timestep at a time by updating the state of the dlnetwork. The predict (Deep Learning Toolbox) function allows you to produce the output prediction, along with the updated network state. This rulPredictAndUpdate function takes in a single-timestep input and updates the state of the network so that subsequent inputs are treated as subsequent timesteps of the same sample. After passing in all timesteps one at a time, the resulting output is the same as if all timesteps were passed in as a single input.

type('rulPredictAndUpdate.m')

function out = rulPredictAndUpdate(in, dataFormat)
%#codegen

% Copyright 2020-2024 The MathWorks, Inc.

persistent mynet;

if isempty(mynet)
    mynet = coder.loadDeepLearningNetwork('rulDlnetwork.mat');
end

% Construct formatted dlarray input
dlIn = dlarray(in, dataFormat);

% Predict and update the state of the network
[dlOut, updatedState] = predict(mynet, dlIn);
mynet.State = updatedState;

out = extractdata(dlOut);

end

Create the input type for the codegen command. Because rulPredictAndUpdate accepts a single timestep data in each call, specify the input type matrixInput to have a fixed sequence length of 1 instead of a variable sequence length.

matrixInput = coder.typeof(single(0),[17 1]);

Run the codegen command to generate PIL based mex function rulPredictAndUpdate_pil on the host platform.

codegen -config cfg rulPredictAndUpdate -args {matrixInput, coder.Constant(dataFormat)} -report

### Connectivity configuration for function 'rulPredictAndUpdate': 'Raspberry Pi'
Location of the generated elf : /home/pi/MATLAB_ws/R2024b/home/yangzhu/Documents/MATLAB/ExampleManager/yangzhu.24bGR2/deeplearning_shared-ex26065676/codegen/lib/rulPredictAndUpdate/pil
Code generation successful: View report

Run Generated PIL MEX Function on Test Data

Run generated MEX function rulPredictAndUpdate_pil for each time step data in the inputData sequence. After you pass all timesteps, one at a time, to the rulPredictAndUpdate function, the resulting output is the same as that in the first approach in which you passed all inputs at once.

sequenceLength = size(inputData,2);
YPred2 = zeros(1, sequenceLength);
for i=1:sequenceLength
    inTimeStep = inputData(:,i);
    YPred2(:, i) = rulPredictAndUpdate_pil(inTimeStep,dataFormat);
end

### Starting application: 'codegen/lib/rulPredictAndUpdate/pil/rulPredictAndUpdate.elf'
    To terminate execution: clear rulPredictAndUpdate_pil
### Launching application rulPredictAndUpdate.elf...

Compare Predictions with Test Data

Use a plot to compare the MEX output data with the test data.

figure('Name', 'Stateful LSTM', 'NumberTitle', 'off');

plot(YValidate{idx},'--')
hold on
plot(YPred2,'.-')
hold off

ylim([0 175])
title("Test Observation " + idx)
xlabel("Time Step")
ylabel("RUL measured in cycles")

Clear PIL

Terminate the PIL execution process.

clear rulPredictAndUpdate_pil;

### Host application produced the following standard output (stdout) and standard error (stderr) messages:

References

[1] Saxena, Abhinav, Kai Goebel, Don Simon, and Neil Eklund. "Damage propagation modeling for aircraft engine run-to-failure simulation." In Prognostics and Health Management, 2008. PHM 2008. International Conference on, pp. 1-9. IEEE, 2008.