How to load a mat. file containing one big matrix
20 views (last 30 days)
Show older comments
Klara Stribrna on 9 Mar 2023
Commented: Walter Roberson on 10 Mar 2023
Hello, I am working with 23.8 GB mat. file so I can't load it with the load function. I've tried to load it with matfile function but I have found out that in only contains one variable, which is a 1x1 struct and is again too big to be loaded. I've tried to read one element of the matrix (it's 4-dimensional) and it didn't work, still the same error ("Requested 994x961x3346x1 (23.8GB) array exceeds maximum array size preference (15.8GB). This might cause MATLAB to become unresponsive.") The mat. file was generated from an external program and I think the problem might be that it was not saved with the -v7.3 flag so it doesn't support partial loading. Is there a way to resave this file with the -v7.3 flag without having to load it all first? Or do you have any other suggestions how to acces this data?
Stephen23 on 10 Mar 2023
"Is there any difference (performance)"
As far as I can recall, if the size limit is unchecked then performance is unaffected (at least, nothing humanly noticeable) until an array larger than the installed memory is requested, then the virtual memory is used and it is slower.
But still quite useable if using an SSD.
Jan on 9 Mar 2023
Edited: Jan on 10 Mar 2023
If the variable in the MAT file has more than 2GB, it must have the v7.3 format.
You try to extract a 994x961x3346 subarray, which needs 25.57 GB for the type double (The difference to 23.8 is ([EDITED], "most likely" removed) a mixing of GB and GiB). But you explain, that the complete file has 23.8 GB also?
If the wanted submatrix does not match into your RAM, this is not a problem of the MAT file, but of an insufficient amount of installed RAM. So install more RAM or import parts of the submatrix only.
Walter Roberson on 10 Mar 2023
If you are talking about RAM, for historical reasons it is very common to use 1024 as the multiplier. For example 8 GB RAM is typically 8 * 1024^3 -- at least when you are talking in terms of how much RAM is installed or what the limits are for linking or program size. But if you are looking at how many kilobytes a program is using in the process monitor, there is a tendancy to slip into 1000 as the prefix.
But when you are talking about hard disks especially, 1000 has used as the multiplier for quite a number of years now; possibly not originally but the marketing department won the battle with hard disks a long time ago.
Find more on Workspace Variables and MAT-Files in Help Center and File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!Start Hunting!