LSTM classification for days

I have 20 control and 20 infected samples. The spectral data of both control and infected samples were collected from day1 to day 15 and saved in excel fromat as day1.xlsx, day2.xlsx........day15.xlsx. For each day the data size is 40*285 (sample*bands). I want to read entire excel data and want to apply LSTM model to classify control and infected samples by considering 15 days as a timepoint. Also I want to know how many days are enough to classify the samples. Any suggestion or related code will be helpful. Thanks!

[영준]

1. Define the problem

Spectral data from control and infected samples were collected over time for 15 days. The goal seems to be to utilize the data to classify whether each sample is a control or an infected sample To do this, we need to learn how the data changes over time, and design a model so that these changes reflect the differences between the control and infected samples We assume that the data is a simple time series, as we need to look closely at the structure of the data to design a model.

2.Data preprocessing

Bring all the data together and sort it into a structure of number of samples, time axis (1-15 days), and spectral bands (285). Each band value is normalized or standardized to clarify differences between spectra. This ensures that scale differences in the data do not affect model training.

3. Model design

The model is designed based on a Long Short-Term Memory (LSTM) network. LSTMs are well-suited to learning time-series data, and we use them to learn the pattern of spectral changes over 15 days for each sample. The size of the cells should be more than the variation or minimum periodicity. If you are looking at the overall behavior of the data and see changes at specific points in time, it may be better to utilize an attentional structure.

4. Evaluate the importance of the time scale

Instead of using all 15 days of data in the model training process, evaluate whether data up to a certain date is sufficient for training. For example, train the model using only data from days 1 through 5, or only data from days 1 through 10, and compare the performance. This allows you to see the minimum date range where model performance converges.

If you are able to generate a model based on LSTM, evaluate its importance, and find some variation in the timing of the changes, we recommend using a mixture of attitudes, otherwise, you may want to start with a modified LSTM or feature engineering.

[영준]

Also, when designing the LSTM, it is recommended to be more specific about whether the questions are for each hour of the 15-day period or for the next hour.

[sensor]

Thanks 영준 for your suggestion. Could you explain more about the second point. "Bring all the data together and sort it into a structure of number of samples, time axis (1-15 days), and spectral bands (285)". It mean that number of samples will be 600 (40*15), days will be 15 and spectral bands will be 285? How can i do this? Do I have to concatenate all the samples? if so in which direaction ?

Answer 1

[Aastha]

Hi @sensor,

I understand that you would like to train an LSTM model to classify samples as either control or infected, which involves a sequence classification task. You may refer to the steps mentioned below to do so:

1. You will need to decide on the number of days (“numDays”), number of samples (“numSamples”), and the number of features or spectral bands (“numBands”). In MATLAB, you can use the “readmatrix” function to load your data from Excel files and organize it accordingly. You may refer to the MATLAB code snippet below for reference:

numDays = 5;
numSamples = 40;
numBands = 285;
data = zeros(numSamples, numBands, numDays);

for day = 1:numDays
    data(:, :, day) = readmatrix(sprintf('day%d.xlsx', day));
end

For any further information on “readmatrix” function, you can refer to the documentation link mentioned below:

https://www.mathworks.com/help/matlab/import_export/import-numeric-data-from-a-text-file.html

2. Rearrange the data to match the LSTM input format using the “permute” function in MATLAB (samples x days x bands).

data = permute(data, [1, 3, 2]); % Resulting size: 40 x 5 x 285

Kindly refer to MathWorks documentation of “permute” function for any queries on it:

https://www.mathworks.com/help/matlab/ref/double.permute.html

3. You can then assign class labels to each sample as illustrated in the MATLAB code snippet below:

labels = [ones(20,1); 2*ones(20,1)];

4. MATLAB expects sequence input as a cell array, with each cell containing a sequence matrix. You can convert your data to this format using the following MATLAB code snippet:

X = cell(numSamples, 1);
for i = 1:numSamples
    X{i} = squeeze(data(i, :, :)); % Each sequence: days x bands
end
Y = categorical(labels); % Convert labels to categorical format for classification

You can now proceed to define and train your LSTM network.

For more details on setting up the network architecture and training, you may refer to the MathWorks documentation on sequence classification using LSTM:

https://www.mathworks.com/help/deeplearning/ug/classify-sequence-data-using-lstm-networks.html

Hope this is helpful!