Very large Matlab data file (q-table)

Hi,

I have a code to run a training of q-learning based model. This code save the q-table and learning variables in a file using this statement:

save([filename '.mat'],... 'trial', 'QT','QTableEnterMap','all_steps','all_accuracy','all_timespent');

I run the code in remote server and after completing the training, I try to download the generated file and it was very large (1.09 GB) and took long time to be downloaded. Then, when I try load the file, I got error that said Error using load Unable to read MAT-file batch1_e.mat. Not a binary MAT-file. Try load -ASCII to read as text.

My questions are: 1- Is it possible to get very large file after during training q-learning model or there is a problem in the code?

2- How to load this very large file?

Thank you.

Answer 1

[ahmed nebli]

1- First question: I'm not sure, but I do think yes it's possible because 1 GB is not a big file compared to other files generated after training.

2- Second question,: There is a function called matfile. This function accesses the data without loading it into the memory (used especially for big data). For further information take a look at this https://www.mathworks.com/help/matlab/ref/matfile.html