Large For Loop Executions

Hello Everybody,

I need to execute a for loop for a matrix (34 x 24) which will result in a matrix with 6e36 rows and 24 columns. Of course when I try to do that, matlab gives me the truncating error.

I know its very huge but I want to know if there is a way to execute that large "for loops"?? And if there is a faster way of course that would be great.

Thanks in advance,

[Adam]

Whether or not there is a faster way probably depends entirely on what you are doing in the loop. Judging from the numbers you give I assume your output is 34^24 in size.

The problem would appear not to be the for loop (although this may be slow), but the memory usage. I would imagine you need to do the result in chunks and save each to file.

What does your for loop index look like? If it is 1:6e36 then certainly that is too large and you need to break it down, if it just just 1:34 then the problem is just with memory and not for loop truncation.

[the cyclist]

I think it would be helpful to post a highly scaled-down version of your problem, ideally with executable code, and than indicate which dimensions you need to scale up.

[Ayman Esmat]

I understand this.. I will give u an example to what I'm trying to do exactly.

Lets say I have x =

     1     4     7
     2     5     8
     3     6     9

I wrote a code that will place the values in a new matrix like this (where every probability for combination of values in each row is placed) 1 4 7 1 4 8 1 4 9 1 5 7 1 5 8

etc....

The result will be 1 4 7 1 4 8 1 4 9 1 5 7 1 5 8 1 5 9 1 6 7 1 6 8 1 6 9 2 4 7 2 4 8 2 4 9 2 5 7 2 5 8 2 5 9 2 6 7 2 6 8 2 6 9 3 4 7 3 4 8 3 4 9 3 5 7 3 5 8 3 5 9 3 6 7 3 6 8 3 6 9

I hope u get what I'm trying to do. What I want here is that instead of matrix being 3 x 3, i want it 34 x 24 to produce the 6e36 rows and 24 columns.

Hope I explained it well.

[Ayman Esmat]

I'm sorry, the result of the matrix should be something like this,

      1     4     7
     1     4     8
     1     4     9
     1     5     7
     1     5     8
     1     5     9
     1     6     7
     1     6     8
     1     6     9
     2     4     7
     2     4     8
     2     4     9
     2     5     7
     2     5     8
     2     5     9
     2     6     7
     2     6     8
     2     6     9
     3     4     7
     3     4     8
     3     4     9
     3     5     7
     3     5     8
     3     5     9
     3     6     7
     3     6     8
     3     6     9
     1     5     8
     1     5     9

[Adam]

Well, as Mohammad Abouali points out below, I really don't see how you can hope to do this on the data size you say - the disk space required is monolithic. There must be massive redundancy in what you are trying to do...to go from 24*36 to data sizes that are bigger than human comprehension cannot be a solution to whatever problem you are trying to solve.

Even if you had enough disk size to store this, no human nor algorithm could hope to make use of such a vast amount of data.

[Ayman Esmat]

Well like I said, for me its all about all the different combinations of these numbers. Further in my code, I'm going to decide which combination I will choose and which I will eliminate. But it seems that my approach to the problem was wrong. I need to rethink it.

Answer 1

[Mohammad Abouali]

How do you want to store that in memory?

6e36*24 elements stored at 1B an element (uint8) requires more than 1.34e29GB of memory. I don't think you can even store that in a hard drive, let alone on memory.

Are you sure about your sizes?

[Ayman Esmat]

unfortunately I'm very sure about the size.

[Ayman Esmat]

I explained what my code is doing above. If u check it u will understand what I'm trying to do.

[Mohammad Abouali]

I suggest instead of producing all those combination and storing it somewhere (which by the way, you won't be able to store it with todays disk sizes even on the largest supercomputer available), try an alternative approach.

Write a function that gives the n-th combination. So, given your example (the 3x3 matrix) if I call myFun(A,1) it gives you 1 4 7 and if you type myFun(A,2) it gives you 1 4 8.

Then you can pool any combination that you want and do your further computation.

[Ayman Esmat]

I'm sorry but u can you further elaborate on this.. I don't understand ur example.

[Mohammad Abouali]

Let's say there is a storage facility that can host that much of data, i.e. 6e36*24 at 1B per element, So let's say you have access to such storage facility. If your write speed is 1GB per second it would take you more than 4.2e21 years to finish writing it to a disk.

[Ayman Esmat]

I see. Well I don't have anything to do so I might as well get started. (Kidding)

Thanks for the help, I will try to figure out another way to approach my problem

[Mohammad Abouali]

So something like this"

A=  [1     4     7; ...
     2     5     8; ...
     3     6     9];

% let's say you want the 8-th combination i.e. row 8 of the output you listed above or [1     6     8]
n=8

% now getting the n-th combination.
[k,j,i]=ind2sub(size(A,1)*ones(1,size(A,2)),n);
[A(i,1) A(j,2) A(k,3)]

in your case which you have 24 columns, instead of i,j,k you need i1,i2, i3, ... i24; and notice how the order is reversed.

[Ayman Esmat]

Thank you very much for this help.. I will try this and get back to you.

Thanks again

[Mohammad Abouali]

you are welcome

[Ayman Esmat]

Dear Sir, I tried ur code and it worked properly. Thank u very much for ur help.

Although, I have a question. In my original code, I constructed the result matrix column by column. And then after the matrix is fully constructed. I start another loop checking the values of each row according to my conditions and omitting the rows that are not feasible to me. (Number of columns doesn't change) Now with ur code, the result could be constructed row by row which is very helpful to me; I can check each row the moment its given to me in the loop and I can decide whether its feasible or not. Matlab won't have to store all the values of the matrix like my old code did, now it will store only the feasible values I want. My question is, since I won't need to store all the row values in case of my huge matrix of 6e36 rows (the 24 columns can't be changed). I only need certain rows which are feasible to me. Can I put the condition of omitting the unwanted rows in ur code so it will store the feasible rows and not store all the possible rows and run the matlab for loop for n=1:6e36 . Can this be done or am I circling in the same dilemma again?

Thanks again for ur help..