How can I calculate very large matrix and store the values?

Suppose I have 1 million-by-1 million matrix.

I will read later about creating very large matrix at here http://blogs.mathworks.com/loren/2011/10/14/new-mat-file-functionality-in-r2011b/

Now here's the problem.

I have 1 million points (with their Euclidean coordinate). I want to calculate distances between them, for example distance between point 1 and 1, 1 and 2, 1 and 3, and so on; distance between point 2 and 1, 2 and 2, 2 and 3, and so on; distance between point 3 and 1, 3 and 2, 3 and 3, 3 and 4 , and so on.

I want to store those calculations in that very large matrix.

Is mapreduce, or any other methods, can do the calculations and store them? If yes, how can I do that?

Any help will be greatly appreciate.

Thank you very much.

Answer 1

[Hugo]

I guess one option would be to reduce the size of your data using clustering such as K-means or Gaussian mixtures, and replace all points in a cluster with the centre of that cluster. You will end up with a reduced distance matrix and a map between cluster centres and data points. Of course, doing so will incur in an error, which can be decreased by increasing the number of clusters, at the cost of increasing the computational expenses.