Finding 67000x67000 size weighting matrix

Hello,

I need to find a large 67000x67000 size weighting matrix for a number of signals I have. For example I have 67000 signals and I want to find similarity of each signals with other signal based on, let's say, correlation coefficient, I use the command

w = corrcoef(all_signals); % all columns are N-element 67000 different vector signals i.e. w is of length Nx67000

if I have let's say 1000 signals then the machine can compute the 1000x1000 w-matrix but for 67000 it gets out of memory. Kindly see the image for reference.

I can use for loops but it'll take days. Any suggestions or tips on how to do it?

Thanks.

[Matt J]

Judging from the error messages, it doesn't look like you have enough memory to hold a 67000x67000 matrix. Can you successfully generate one more simply, e.g., as follows?

>> r = rand(67000);

[Muzammil Behzad]

I shared the error message in my post.

Nope I can't even generate what you asked. I get the following error:

"Error using rand Out of memory. Type HELP MEMORY for your options."

What can I do?

[Walter Roberson]

What are you going to do with the large array produced? Could that be done incrementally, perhaps updating statistics after every few columns? That could at least allow you complete the operation. The alternative is to add memory, lots of memory.

[Muzammil Behzad]

I need to process the signals later. However, as you said, is there any way that let's say for each signal out of 67000 signals, I just find the similarity with the adjacent 10000 signals instead of all signals. For example, for signal number 1, I'll find the similarity with starting from signal in column 1 till signal in column number 10000. Similarly for lets say signal at column number 15000, I'll find the similarity with signal in column 10000 till signal in column 20000. This will reduce the cost but is there any way to do it efficiency for all 67000 signals? The w-matrix will become of the size 67000x10000.