How to improve LSTM algorithm to extract features of time derivative signal, GPU, deep learning

Hi I am working on deep learning tool to desing a technique to classify fault type, im facing difficulty in preparing data type and I dont know why please have a look at my code and advice.

%unamed is 170*244 double type matrix
Daten = unnamed;
[m,n] = size(Daten) ;
%Split into train and test
P = 0.7;
Training = Daten(1:round(P*m),:) ; 
Testing = Daten(round(P*m)+1:end,:);
%XTrain is 119*243 double type matrix
XTrain = Training(:,1:n-1);
%YTrain is 119*1 double type matrix
YTrain = Training(:,n);
%XTest is 51*243 double type matrix
XTest = Testing(:,1:n-1);
%YTest is 51*1 double type matrix
YTest = Testing(:,n);

layers = [ ...
    sequenceInputLayer(119)
    lstmLayer(100,"OutputMode","sequence")
    dropoutLayer(0.1)
    lstmLayer(100,"OutputMode","last")
    fullyConnectedLayer(1)
    softmaxLayer
    classificationLayer];


options = trainingOptions("adam", ...
    "MaxEpochs",150, ...
    "MiniBatchSize",miniBatchSize, ...
    "Plots","training-progress", ...
    "Verbose",false, ...
    "Shuffle","every-epoch", ...
    "LearnRateSchedule","piecewise", ...
    "LearnRateDropFactor",0.1, ...
    "LearnRateDropPeriod",20,...
    'ValidationData',{XTest,categorical(YTest)});

net = trainNetwork( XTrain , categorical(YTrain) , layers , options);

Answer 1

[Walter Roberson]

You are splitting up your data by rows, implying that each row is a sample and that the number of features is the one fewer than the number of columns, with the last column being the class.

That is fine as far as it goes, but when you train, the features must go down columns, not across rows.

    sequenceInputLayer(119)

That says that you have 119 features, but you do not: you have extracted 119 samples.

You need to transpose, like

XTest = Testing(:,1:n-1) .';

and you need to use

    sequenceInputLayer(NumberOfFeatures)

Now, when you use sequenceInputLayer, what you have to pass in as training data is a cell array. The number of entries in the cell array must match the number of entries in the targets you pass in. Each individual cell entry must be an array in which the number of rows is the same as the number of features, and the number of columns is the same as the number of time steps for the sequence -- so each row corresponds to all the time steps for one particular feature.

It is not clear to me that you really do have sequence data.

[zain yousaf]

i am taking fft of the time signal and using it to identify fault classification, though results are not satisfactory, any change in deep learning can help?

i have found the solution of the mismatch of the arrays but i cannot find solution to improve deep learning layers to optimze my results. any feedback?

[Walter Roberson]

How many features are you working with? The number of features needs to be the same for each sample.

In your input array, which is a 2D double() matrix, how do you mark where the end of each sequence is or the end of each feature is?

You implicitly have 3 dimensions of data:

• you have features
• you have several datapoints for the features, with each datapoint representing a different time the feature was measured at
• the two together are a 2D array with the rows corresponding to different features and the columns corresponding to different times
• you have groups of those 2D arrays, one cell entry for each group. Each group element corresponds to a particular target.

But... your input data is 2D, with the last column being the target, which requires that each row must be broken up into multiple segments to be reconstructed into a 2D array to be inserted into cell entries for use with sequenceInputLayer. How should it be broken up? For example do you happen to have exactly 5 features, and the columns without the target information should be reshape()'d to give you (say) 5 features at each of 14 time points?

I am not convinced that you really have a "sequence" input. What I suspect is that you have time points, and that you want to fft that, and that you want to pick out particular bins from the result and use those as the features. Or perhaps you want to declare that every output bin is a feature and let it figure out which ones have relevance. Do keep in mind, though, that the result of fft is very likely to be complex valued, and deep learning is usually only done on real values.

[zain yousaf]

Sir you are right, I am taking FFT of the time signal and then abs function applied to utilize real values, I am using those real values as my input samples for 2ms time, I am not using time signal in it, i am just using using the y-axis point of the real values of the FFT, then im fitting those real yaxis point into my 2d array with last col as the class of that sample. All samples are same in lenght. Do you think, im doing it right?, if not please advice as it can help other in this area of work as well. What should i do to improve classification results as LSTM is failing.

[Walter Roberson]

Ah, in that case after you have processed through to abs(fft()) then that should be one sample and the length of that should be the number of features, and you should not be using sequenceInputLayer

[zain yousaf]

what should i use sir? as there should be some way to give input to the deep learning layer. please adivce.

[zain yousaf]

it has only four ways to take input, imageInputLayer,image3dInputLayer, sequenceInputLayer and roiInputLayer which way is use to give my input to the deep learning layer, please advice as it can help to solve my error.

[Walter Roberson]

I guess you could use a cell array containing a column vector that was the output of abs(fft(data)), with the number of features set to the length of the data.

However if your input is real-valued then abs() of the fft() is going to have duplicate values in it, as fft() of real data results in a conjugate reverse:

[f0, f1, f2, f3, f4, ... , f4', f3', f2', f1']

so you should consider only using the first half of the abs() as features.

[zain yousaf]

Sir First of all very thankful for your answer, this means alot. Sir we are doing the same thing rightnow, we are using cell array that contains column vectors of abs(fft(data)), but sir the accuracy of the training is 60% and testing accuracy is 50%, i even applied mean and std function as shown below, but still little improvements

M=mean(XTest);
>> S=std(XTest);
>> Z=((XTest - M)./S);
>> XTest1=num2cell(Z,1);
>> M=mean(XTrain);
>> S=std(XTrain);
>> Z=((XTrain - M)./S);
>> XTrain1=num2cell(Z,1);
>> M=mean(ZTest);
>> S=std(ZTest);
>> Z=((ZTest - M)./S);
>> ZTest1=num2cell(Z,1);


My layers are 
layers = [ ...
    sequenceInputLayer(243)
    lstmLayer(100,"OutputMode","sequence")
    dropoutLayer(0.1)
    lstmLayer(100,"OutputMode","last")
    fullyConnectedLayer(3)
    softmaxLayer
    classificationLayer];
options = trainingOptions('adam', ...
            'InitialLearnRate',optVars.InitialLearnRate, ...
             'MaxEpochs',50, ...
            'LearnRateSchedule','piecewise', ...
            'LearnRateDropPeriod',40, ...
            'LearnRateDropFactor',0.1, ...
            'MiniBatchSize',miniBatchSize, ...
            'L2Regularization',optVars.L2Regularization, ...
            'Shuffle','every-epoch', ...
            'Verbose',false, ...
            'Plots','training-progress', ...
            'ValidationData',{XTest1,YTest1}, ...
            'ValidationFrequency',validationFrequency);

[Walter Roberson]

Your results would suggest to me that your abs(fft()) of the data you have collected, is not useful information as features for classifying faults -- at least not with that sequence of layers.

I do not have any experience with deep learning, so I am not able to suggest any other sequence of layers.

I would point out to you that you only have one time step, but that lstmLayer has to do with relating timesteps within the same sequence, and therefore it does not make sense to use those layers for your kind of data.

[zain yousaf]

what sir if I put direct voltage or current value, is that going to help me or not? for example if I use 0.1 ms voltage or current values direclty into inputsequence layer, that going to improve or not, what if I add current and voltage values along time steps of 0.1ms? that can help? I mean that if I add time steps which is order of 0.1ms along my voltage or current sample, that make sense?

[Walter Roberson]

if I put direct voltage or current value, is that going to help me or not?

not . Adding direct current would shift the mean of the signal, and would affect only the first bin of the fft() result. However, you are subtracting out the mean, and that would have the effect of getting rid of the DC offset. So NO, it will not help.

for example if I use 0.1 ms voltage or current values direclty into inputsequence layer, that going to improve or not

not

what if I add current and voltage values along time steps of 0.1ms

I doubt that would help.

I mean that if I add time steps which is order of 0.1ms along my voltage or current sample, that make sense?

It does not make sense to me that you would want to do that. Even if you were not subtracting out the mean, the only effect on the fft would be to change the first bin, that records the mean value. It is not going to help.

In order to use lstm usefully, you need to have recordings of your signal evolving in time. That might require that you record your signal for longer but break it up into pieces to monitor the evolution of the frequency distribution, something like a spectrogram effect.

Depending on what you are doing, in some cases it might make sense to provide a series of stimuli, all a bit different from each other, in order to see the response to that stimuli. I suppose in some cases, increasing the voltage or current in steps could provide sufficiently varying stimuli, if there was reason to believe that the system would react differently. However if you were looking for something like resonance frequencies, I would expect that you would get more information out of different modulating waveforms.

[zain yousaf]

what if sir I use CNN layer, instead of LSTM, CNN has the potential to extract the unique features and decide on it, can i do that?

[Walter Roberson]

That might be better.

[zain yousaf]

but sir how i can use my data set along CNN, all CNN examples are for image classification, it gives the impression that CNN is only for image, there is no example availible that shows us that it possbile to use it for my work, any help regarding that will be very useful sir.