Training feedforward network with multiple datasets

Introduction

I am working on battery State of Charge (SOC) estimation for my project. I am currently working with 3 battery types (Li-Ion, Li-Po, and Ni-MH) each having a capacity of 1000mAh. During my battery characterization I discharge these at different temperature (5C, 15C, 25C, 35C, and 45C) and at various load current (30mA, 50mA, and 100mA). So, naturally I get 45 log files (3 batteries x 5 temperatures x 3 currents).

Each log file (a csv file in this case) represents battery discharge characteristics for respective current and temperatures.

The parameters that are stored in my log files include (voltage, current, temperature, and SOC).

Input Parameter = Voltage, Current, Temperature

Target = SOC

Now, I designed a feedforward neural network (non-linear regression based) to estimate the SOC and it is working fine for one log (For Example, Li-Ion battery discharge at 5C and 30mA current).

Problem

1. I would like to test whether my model works well for dataset of the same battery
2. Whether the model also works for dataset of other batteries as well

Initial Guess

I cannot append the values of above input parameters into file, as the input parameters may give me duplicate values and dropping these values may result in poor prediction.

Optimal Guess

In my opinion, the most suitable strategy would be to Train and Evaluate the designed network for each set of CSV file and record its performance. However, this approach is non-intuitive and requires calling all csv files one-by-one and run the model manually for each such iteration.

I was wondering if there was a better solution to address this problem? I looked into creating a datastore (that will contain all of my CSV files), but I do not know how to use this datastore to train and evaluate the model.

Can someone please help me to create a datastore for (lets say 45 csv files) and how to use them one-by-one to train/evaluate the model?

Thanks

[osman alper altun]

Hello @Omer Ali,

I'm dealing with almost the same issue for my master. Do you have any solution?

Best Regards,

Answer 1

[Sai Pavan]

Hello Omer,

I understand that you want to train a feedforward neural network with 45 csv files separately with the help of datastore.

To address the problem of training and evaluating a neural network model across multiple datasets, using a datastore is indeed a practical approach. We can create a datastore that points to your CSV files, and then iteratively read batches of data for training and evaluation. This approach allows us to systematically train and evaluate your model across multiple datasets without manually handling each file and also keeps the workflow organized and scalable to more data files if needed.

Please refer to the below workflow to accomplish the training and evaluation of the neural network with multiple datasets separately:

• Create a datastore for the csv files
• Prepare the neural network
• Loop through each csv file for training and evaluation

The code snippet shown below illustrates the process:

% Path to the directory containing your CSV files
folderPath = './battery_data/';

% Create a datastore for all CSV files in the directory
ds = datastore(folderPath, 'Type', 'file', 'FileExtensions', '.csv');

% Loop through each file in the datastore
while hasdata(ds)
    % Read the next file
    dataFile = read(ds);
    
    % Load the CSV file (assuming the first row contains headers)
    dataTable = readtable(dataFile{1,1});
    
    % Assuming the columns are named 'Voltage', 'Current', 'Temperature', 'SOC'
    inputs = dataTable{:, {'Voltage', 'Current', 'Temperature'}};
    targets = dataTable{:, 'SOC'};
    % Split the data into training and validation sets
    % [trainInputs, trainTargets, valInputs, valTargets] = splitData(inputs, targets);
    % Create and train a neural network model with the data
    % model = trainNetwork(trainInputs, trainTargets, layers, options);
    % Evaluate the model on validation set
    % predictions = model.predict(valInputs);
    % Calculate performance metrics
end

Please refer to the below documentation to learn more about:

• "datastore" function: https://www.mathworks.com/help/matlab/ref/datastore.html
• "hasdata" function: https://www.mathworks.com/help/matlab/ref/matlab.io.datastore.hasdata.html
• "read" function: https://www.mathworks.com/help/matlab/ref/matlab.io.datastore.read.html
• "readtable" function: https://www.mathworks.com/help/matlab/ref/readtable.html

Hope it helps!