Best way to save big matrices

I'm looking for the best way to save big (both full and sparse) matrices in files. What I need to be able to do is:

- Read only certain entries from the file (certain matrix entries, that is)

- Create a sparse matrix in MATLAB from this file without creating a full matrix first (assuming the file contains a sparse matrix, of course)

The reason is quite simple: The nonsparse matrices will be too big to be stored into memory as a full matrix.

I read about the CDF file format which I'm not yet familiar with, so I don't know if it's possible to create a sparse matrix in Matlab from a CDF File?

[Ingo]

A little elaboration:

Given are n data points and now I want to calculate the (n x n) matrix containing the pairwise distance between all points. This is a full matrix, too big to store it in memory.

The next step would be to make this matrix sparse by, for example, ignoring all values less than a certain treshold. This sparse matrix can be stored in memory.

Answer 1

[the cyclist]

The most recent versions of MATLAB have the matfile() command, which I believe does at least some of what you want.

[Ingo]

This seems like the perfect solution. I'm gonna give it a try as soon as possible. Thank you very much!

[the cyclist]

If you do find it works out for you, please "accept" the solution, which may help future users with similar problems.

[Ingo]

Hi. Sorry for being late on that!

Yes, matfile works for what I intended to do. However, for my purposes it is just way too slow. I'll have to stick with smaller data sets and try to create the resulting sparse matrices without ever calculating full matrices.

However, matfile() still does solve the problem.

Thanks again and Greetings

Ingo