Iteratively save data to mat file
13 visualizzazioni (ultimi 30 giorni)
Mostra commenti meno recenti
I have to read, process and plot data from 700 text files. I want to save the processed data from one text file to a Mat file before moving on to the next text file. This is a precaution in case of error. I would prefer that all the data is saved to a single variable in a single Mat file but I don't see how to do that efficiently.
If I use save('myMatFile.m','x') to save the data from each file, then I will need to create 700 variables in the Mat file; not ideal. If I append a single variable in the Mat file then Matlab partially loads it each time; also not ideal. Originally, I use dlmwrite to export to a text file but that is slow.
Also, I'd like to know if my approach is wrong and there is a better way to do this. ANYTHING to speed up this incredibly slow IO process.
0 Commenti
Risposte (2)
Stephen23
il 11 Ott 2018
Modificato: Stephen23
il 11 Ott 2018
"I would prefer that all the data is saved to a single variable in a single Mat file but I don't see how to do that efficiently."
"If I append a single variable in the Mat file then Matlab partially loads it each time; also not ideal."
2 Commenti
Walter Roberson
il 11 Ott 2018
Consider that if you were changing an existing variable, then MATLAB would have to locate the variable in the file, figure out its size, and make a decision about whether to overwrite in place or to delete the variable and write a new one.
When you append a new variable, MATLAB must first search the .mat file to determine that the variable is not already present, after which it makes whatever changes it needs to the .mat file to write the new variable.
The only way that these processes could be done without partially reading the .mat file would be if every change made by matFile were by simply writing new information on the end without looking inside the file to see what was already there. Not impossible, but it would have the consequence that reading the .mat file would require scanning from the beginning to the end for everything, just in case there was an update later on in the file that overwrote the existing information. It could be done that way, but it would make every read inefficient.
gonzalo Mier
il 10 Ott 2018
First, use the -nocompression option for the save operator if you have a lot of info.
Maybe you could improve your performance writing the info in 700 different mat files as save("mat_"+num2str(i)+".m") so you don't have to read the info each time.
1 Commento
Walter Roberson
il 11 Ott 2018
I recently posted a timing test using matFile with and without compression. For the rand() that the user was working with, turning off compression made a big difference. It is plausible that in some other cases like a mostly 0 matrix that compression might be more efficient due to reduced disk i/o.
Be sure to save with -v7.3 to use matFile efficiently
Vedere anche
Categorie
Scopri di più su Workspace Variables and MAT-Files in Help Center e File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!