read file variables methods
Mostra commenti meno recenti
I got a large data matrix (3Gb) of signal data which I saved to a .mat file. Since the data size is large I figured out it will be wiser to access the file and read each time a block of data, process it and move to the next block instead of loading the complete file. I'm wondering should I use the matfile method, fopen and fread or should I use a datastore and read command. What is faster from the processing point of view and easier to work with? Or any other considerations? Thanks
3 Commenti
per isakson
il 11 Nov 2016
"data set"   is that one numerical array?
"fairly large data set"   how large compared to the physical memory?
Talmon Alexandri
il 12 Nov 2016
per isakson
il 14 Nov 2016
Given
- "large data matrix (3Gb) of signal data which I saved to a .mat file"
- "Total RAM size - 3G[B]"
- "[one] plain numerical array 10 million lines by 48 columns"
I assume the mat-file is version 7.3, because that's needed to store an array of that size.
I would say this a a perfect case for matfile, Access and change variables directly in MAT-files, without loading into memory.
It's the easiest to work with and I don't think there is any other way that is significantly faster.
Risposte (1)
Walter Roberson
il 11 Nov 2016
0 voti
Categorie
Scopri di più su Large Files and Big Data in Centro assistenza e File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!