problem with parfor loop

I have the following code

 parfor edge = 1:1000

    var_node = edge_2_var_node(edge, num_of_variable_nodes)  
    check_node = ceil(edge_perm(edge)/dc);                                              

    H(check_node,var_node) = 1;    

 end

but when i try to run it, i have this "The variable H in a parfor cannot be classified." Could anyone help me? Thanks

Answer 1

[Walter Roberson]

You will have difficulty with that set-up. parfor has to assume the possibility that two different edge_2_var_node lookups might give back the same value, leading to the possibility that different parfor cycles might be trying to write into the same H() location.

Is edge_2_var_node a function or an array? Are the edge_perm() values certain to be different by more than dc so that check_node values are certain to be unique? What can you give us that would allow it to be proven that no two H(check_node,var_node) references will be to the same location ?

[Walter Roberson]

Unless parfor can determine that the code will only write to any one location at most once, it will not permit the code to be run in parallel. (Other than some deliberate overrides such as assignin('base'); I do not know why those are allowed.)

Generally speaking, in such a case, (A) try harder to vectorize instead of using parfor; and (B) build independent slices of results that can be combined down to a single slice after the parfor. For example your code appears to be XOR'ing at locations, so create vectors that are normally 0 but 1 at each location to be affected, and after the parfor XOR the vectors together to determine which locations need to be affected.

[freebil]

Thanks for the answer. I vectorized the var_node and check_node variables and i have the parfor loop:

 parfor edge = 1:sum(num_of_edges)
      var_node(edge) = edge_2_var_node(edge, num_of_variable_nodes, num_of_edges, degrees);
      check_node(edge) = ceil(edge_perm(edge)/dc);
 end

Now, i need this loop

 for edge = 1:sum(num_of_edges)
    if (H(check_node(edge),var_node(edge)) == 0)       
       H(var_node(edge),check_node(edge)) = 1;       
    else                                   
        H(check_node(edge),var_node(edge)) = 0;             
    end

 end

but again i can't parfor it. Could you help me? Thanks

[Walter Roberson]

To check: as you go through, if the same H location is encountered more than once, you want to be toggling the content?

[freebil]

I think yes. Is it possible?

[Walter Roberson]

In parallel... I don't think so, but I am not completely positive. As I wrote above "independent slices of results that can be combined down to a single slice after the parfor."

Once you have built check_node and var_node as vectors, then

TH = accumarray([check_node(:), var_node(:)], true, size(H), @xor, false)

will give you an array the size of H in which entries are true only if there were an odd number of toggles to the location. This presumes that H is only 0's and 1's before the initial parfor. If H was completely 0 before the parfor then TH will be the new state of H; if H was not completely 0 before the parfor, the H = xor(H, TH) will be the new state of H. (Unless, that is, H had some entries that were not 0 or 1.)

[freebil]

Oh, really sorry. The loop i need to par for is not the previous. I have to par for this:

 for edge = 1:sum(num_of_edges)
    if (H(check_node(edge),var_node(edge)) == 0)       
       H(check_node(edge),var_node(edge)) = 1;       
    else                                   
        H(check_node(edge),var_node(edge)) = 0;             
    end
 end

[Walter Roberson]

Yes, I know. And I don't think you can do that, but I am not completely positive. I suggest that instead of using parfor for that loop, you replace the loop with the single accumarray() call that I showed, followed by whatever fixup is necessary to get from TH to H according to your knowledge of what state H would have been in before that point.

[freebil]

Really thanks for the answer, but when i run it, matlab gives me the error:

 Error using xor
Not enough input arguments.

 Error in gen_ir (line 72)
 TH = accumarray([check_node(:), var_node(:)], true, size(H), @xor, false);

[Walter Roberson]

Hmmm. Okay, this then:

TH = mod( accumarray([check_node(:), var_node(:)], 1, size(H)), 2);

[freebil]

Thank you very much. This seems to do what I want but now I have a problem with memory because the edges are too many for matlab to run. I want to run it with 10^6 edges. Is it possible?

[Walter Roberson]

What are you maximum var_node and check_node values? In other words, the target size(H) ? I am trying to estimate the density of "touched" locations (distinct locations occupied at some point, even if they end up at 0), and the density of locations that will end up as non-zero.

Before this loop is run, is H completely 0, or are there some non-zero locations in it?

[freebil]

Thanks for the answer. The maximum var_node value is 500000 and the maximum check_node value is 250000. Before this loop run H is sparse with zero elements and its size is 250000x500000. I appreciate your help.

[Walter Roberson]

sparse with zero elements -- so nothing in it, right? In that case you need not create it ahead of time, and can instead use

H = mod( accumarray([check_node(:), var_node(:)], 1, size(H), [], [], true), 2);

You might also want to experiment with

H = accumarray([check_node(:), var_node(:)], 1, size(H), @(X) mod(sum(X),2), [], true);

I would expect this to turn out to be slower because of the many calls to the anonymous function, but at the same time it might end up using less temporary memory.

Answer 2

[freebil]

Thanks for your answer. Actually, the code is

 for edge = 1:sum(num_of_edges)            
    var_node = edge_2_var_node(edge, num_of_variable_nodes, num_of_edges, degrees);    
    check_node = ceil(edge_perm(edge)/dc);                                             

    if (H(check_node,var_node) == 0)      
        H(check_node,var_node) = 1;       
    else                                   
        H(check_node,var_node) = 0;        
    end

 end

And yes, there is possibility that (check_node,var_node) is the same more than 1 time. Is there any possibility to parfor this loop?