Deep Network Designer - LSTM Training Data Post Processing and Creating a Datastore

Hi,

I have been using the shipping example LSTM ROM as the base for my code to create a ROM using an LSTM.

Now that I have the code and network running OK, I would now like to use the Deep Network Designer app and the Experiment Manager to speed up optimisation of the network by running though some different hyperparameters rather than manually changing and re-running the code.

Using the code in the example the training data (dataTrain: with 9 observations and 5 signals) and test data (dataTest: 1 observation with 5 signals) are now contained in a 1x9 cell and 1x1 cell respectively. I have created a signal data store for both cells to allow the data to be imported into the DND app. However, I am encountering an error when I try to train the network. Is the data now pre-processed using the example code the correct format for using the app?

Code modified from LSTM ROM shipping example to partition and format the 10 imported data sets into 9 training data sets and 1 test data set.

% 10 ANSYS Simulations Conducted
numObservations = 10; 

% CR Sweeps
CR_Sweep = [21.0,20.5,19.5,19.0,18.0...
            17.5,17.0,16.5,16.0,15.5]; 

% Concatenate all 10 data sets to a single cell array
data = {data_01; data_02; data_03; data_04; data_05; data_06; data_07; data_08; data_09; data_10}; 

%Prepare Data for Training

%Specify a sample time for Training
downsamplefactor = 1;
sampleTime       = timeInterval*downsamplefactor;     % Sample rate for LSTM Training [s]

%Downsample the training data by extracting time steps with a fixed interval given by the sample time divided by the time interval of the simulation.
intervalDownsampled = sampleTime / timeInterval;
timeStepsDownsampled = 1:intervalDownsampled:numTimeSteps;

for i = 1:numObservations
    dataDownsampled{i} = data{i}(:,timeStepsDownsampled);
end

%Partition the training data into training and test partitions using the trainingPartitions function. 
[idxTrain,idxTest] = trainingPartitions(numObservations,[0.9 0.1]);

CRSweepTrain = CR_Sweep(idxTrain);
CRSweepTest = CR_Sweep(idxTest);
dataTrain = dataDownsampled(idxTrain);
dataTest = dataDownsampled(idxTest);

%Create signal datastore for import into Deep Network Designer
dsd_Train = signalDatastore(dataTrain);
dsd_Test  = signalDatastore(dataTest);

[PB75]

Hi,

Any suggestions on the above query?

Thanks,

Patrick

Answer 1

[Avadhoot]

Hi Patrick,

The error you encountered is caused because you passed dsd_Test as your validation data, and MATLAB requires at least 2 columns in the validation data. Since dsd_Test is created from the file dataTest, it only has a 1x1 cell as its contents.

This issue can be resolved in two ways:

1) While performing the data partition into testing and training sets, set the proportion of test data to anything greater than 0.1. For example:

[idxTrain, idxTest] = trainingPartitions(numObservations, [0.8 0.2]); 

This ensures that dataTest contains at least 2 columns and resolves the error.

2) Assuming that the file dataTest contains a 5x3091 double matrix, as contained in the training set, you can pass the matrix instead of the cell array as the validation data. MATLAB allows the validation data to be a matrix as well. This approach would solve your problem. To do this, follow these steps:

1. Create a variable testDataMatrix.

 testDataMatrix = dataTest{1,1}; 

2. Once this variable is created, you can select it in the "Validation data" drop-down list.

For programmatically implementing this, please refer to the following resource: Options for training deep learning neural network - MATLAB trainingOptions - MathWorks India

I hope this helps.

Regards,

Avadhoot