"Error using zeros" maximum variable size allowed by the program is exceeded

I was trying to generate a 2^57 x 57 zeros matrix and error occurred. I tried with sparse and the error still persist. Is there anywhere can I generate the matrix efficiently.

Please help. I am student, still exploring how to work with MATLAB. Thank you.

Answer 1

[Steven Lord]

If you wanted to create that matrix as a logical array, it would require:

numElements = (2^57)*57;
exabytes = numElements/(1000^6)

exabytes =
        8.21456572032379

To put that into perspective, take a look at the Wikipedia page for exabyte. One comparison:

"Therefore, one exabyte could hold a hundred thousand times the printed material, or 500 to 3000 times all content of the Library of Congress."

To process all possible subsets of a set with 57 elements (which is what I assume you're trying to do, based on the dimensions of the matrix you're trying to create) would require quite a bit of time.

numCombinations = 2^57;
processingTime = years(seconds(numCombinations))

processingTime =
              4.5668e+09

You probably want to rethink your approach, unless perhaps you have a VERY large cluster of machines available.

[Cassandra Wong]

I guess I have to change my approach then. Anyway, thank you sir you the explanation.

Answer 2

[James Tursa]

A matrix of size 2^57 x 57 doubles would take about 6.57e19 bytes of memory ... way way more than your computer has available. So no, there is no way to generate this matrix on your computer. Even a sparse matrix of this size would take too much memory just to store the index data.

[Cassandra Wong]

Thank you sir for the reply.

Answer 3

[Walter Roberson]

A full array for that would require a virtual memory space more than 3 1/2 times larger than can be addressed with 64 bit memory. I don't think you are going to be able to create something that big. You might not be able to create anything that requires more than 2^48 bytes as all current implementations of the x64 architecture only have at most 48 address pins.

[Cassandra Wong]

Thank you sir for the explanation.

Answer 4

[Najme Behdad]

A question comes to me, Is there a way processing for binary large scale matrix in MATLAB 2 ^ 40. Was this possibility added in MATLAB 2016? please help me

[John D'Errico]

Please don't ask questions as an answer. Anyway, the answer is a flat out no. An array with any dimension of 2^40 is still wildly too large of an array to store on your computer.

[Walter Roberson]

I have to disagree. An array with 2^40 double precision elements would occupy only 8 terabytes of memory. That is possible on a sufficiently large system. For example SGI makes servers with up to 64 terabytes of memory. If you have a large enough system, then MATLAB can create an array this large.

You would probably want to create the array as uninitialized

This is a different situation than the 2^57 by 57 that was asked about originally, as that exceeded the address space available to the x64 architecture. 2^40 does not.

Even 2^40 x 40 uint8() or logical() does not exceed 64 terabytes (it would require 40 terabytes.) 2^40 x 40 double() would exceed 64 terabytes though, so you would have to find a really big computer for that. Not impossible, though.

[John D'Errico]

True. But on almost all current systems, 2^40 is far too large. And if someone is asking this question, it is most likely true that they don't understand why that is a problem, so they surely don't have experience on and access to such a large machine. This is surely true for >99% of all MATLAB users.

It is also true that most people who want to do something like this really don't want to do it anyway. They think they do, due to inexperience with computing. Much of the time there are better ways to solve a problem than just throwing huge amounts of memory at it.

[Walter Roberson]

8 terabytes barely gets you onto the Top 500 Supercomputers list (it is more than the computer at current position 499 but much less than the one at current position 494, with memory sizes not given for 495-498). I thus have to doubt ">99% of all MATLAB users". 98% maybe ;-)

I do find, though, that people who ask these kinds of questions and who will actually have the resources tend to specifically mention having the resources, such as if they are cross-checking their planning for a large computer proposal.

[Najme Behdad]

thanks a lot,   What is the solution for the processing of this matrix? 40 is the number of features. Subset = dec2bin (0: 2 ^ 40) == '1'   Do you recommend I use the function "map and reduce"?

[Najme Behdad]

What is the maximum size of binary matrix, Processable in MATLAB 2016؟

[Walter Roberson]

The maximum size of a matrix of logical or uint8, of size 2^N by N, that can be processed in MATLAB, is 2^42 by 42. That requires a system with 168 terabytes of memory.

"What is the solution for the processing of this matrix? 40 is the number of features."

You have not described the goal. It sounds as if you are thinking you need to do an exhaustive trial of all the possible combinations of features being present or not, but what are you calculating?

Considering your use of the word "feature", I suspect you might be trying to find the "best" set of features for some kind of classification. That is a topic known as "feature selection". A complete search is probably not going to do a significantly better job than using some approximations: you would run a significant risk of over-fitting -- the answer you came up with would likely involve small statistical accidents that made that particular choice work marginally better on your test data but worse than other possibilities on real-world data.

There are a number of different approaches to feature selection; different approaches have been studied more with respect to different classification schemes.

In some cases a good approach is to run a number of different random approaches, pick the best several, and use an ensemble classification scheme

[Najme Behdad]

"You have not described the goal. It sounds as if you are thinking you need to do an exhaustive trial of all the possible combinations of features being present or not " Yes, that's exactly right، I want to gain all combinations of features. "but what are you calculating?"

I want to calculate the value of each combination of features. When I know the value of some combinations feature that is equal to one.

[Walter Roberson]

Do you mean you have a vector of 40 values and you are trying to find a subset of the values that totals exactly 1.0 ? If so, then you have to be very careful about round-off in your calculations.

Finding a subset of values with a particular total is known as the Knapsack problem.

https://www.mathworks.com/matlabcentral/fileexchange/22783-0-1-knapsack

http://enthumema.blogspot.ca/2012/09/knapsack-problem-in-dynamic-programming.html

https://labs.ise.ncsu.edu/fangroup/wp-content/uploads/sites/14/2016/08/TA-Lecture-MATLAB.pdf

http://www.matarka.hu/koz/ISSN_1789-2198/vol_7_no_1_2013_eng/ISSN_1789-2198_vol_7_no_1_2013_eng_013-020.pdf

[Najme Behdad]

thank you