Changing large matrices by not completely loading them into memory

Hi,

I'm attempting to modify very large matrices (single, 50e3 x 50e3), which don't make sense to load into the memory. I was wondering what you could recommend me as a data handling strategy? I thought ideally I could always load a let us say 100x100 square modify it and write it back. My working machine uses a SSD connected via M2 so it should be relatively speedy (however of course not nearly as fast as RAM). What suggestions do you have?

Thanks,

Moritz

Answer 1

[Stephen23]

You should read TMW's own advice on working with big data:

http://www.mathworks.com/discovery/big-data-matlab.html

And in particular you might find memmapfile to be of significant interest to you:

http://www.mathworks.com/help/matlab/ref/memmapfile.html

[Walter Roberson]

Or instead of memmapfile, save the .mat with -v7.3 and then use matFile objects to read in portions of the array.

Answer 2

[Alessandro]

Did you check the sparse command out?

[Moritz]

Yes I did. However, I believe this only works if you have a considerable amount of zero elements. In my case however, the amount of zero elements are < 5%.