Saving and resuming the parfor states

Is it possible to save the states of parfor loop, stop/break and restart from the same state. For example, in may case some time Windows crashes and I can not run the simulaiton from the same state, I have to run from initial level. In a simple for loop, I can save the variable i and ourval then I can run the for loop. In the code shown below, I have only variable i chaning in the parfor. Can I save parfor states after some interval and after restarting Matlab I can have all the variables in the wokspace resume parfor from same states.

% cubeL=51;
% XYZw=[90 100 108];
% P_labs, xyz nx3 matrix
% some_function is a function
ourval=zeros(size(xyz));

parfor i=1:cubeL*cubeL*cubeL
    ourval(i,:)= some_function(xyz(i,:),P_labs,rgb,XYZw);

endd 

Answer 1

[Nicolas B.]

Hi Imran,

I would suggest 2 solutions:

1. at each iteration of your parfor loop, you log your progress into a file and then, if MATLAB or Windows crashes, you read that file.
2. you use the DataQueue class to send back data to your main MATLAB process and you log the information in a file (see this documentation).

Does it helps you?

[Abdulllah]

HI Nicolas,

Thank you for reply and time.

1. Using the first approach

I can not use the first approach becasue, I also have to save "ourval" variable. As we can not save values within a parfor so I tried saving within a function. For this purpose, I had to pass the varible "ourval" to the function. parfor does not allow to pass the this varible.

isave=zeros(1,cubeL*cubeL*cubeL);
parfor i=1:cubeL*cubeL*cubeL
    ourval(i,:)= some_function(xyz(i,:),P_labs,rgb,XYZw,isave,i,cubeL,method,ourval);
end

porfor error: can not run parfor due the due to the way the variable ourval is used. in "some_function" I check if the varialbe "i" is already present in the "isave" then I return otherwise I do the task and added the "i" to "isave" to save the "isave" and "ourval". This was good idea but I can not send "ourval" within parfor.

function RGB= some_function(XYZ,P_labs,rgb,XYZw,isave,i,cubeL,method,ourval)

if ismember(isave,i)
    return;
end
%;
%;actual taks
%;
if (i/100==0)
    save('sony_xyzdatai_51crm.mat','ourval','XYZw','cubeL','method') %save invserse data file
    isave(i)=i;
    save isave

 end

end

2. Using the second approach

I am not yet able to do the second approach. Is it possible that you guide further

q = parallel.pool.DataQueue;
afterEach(q, @disp);
isave=zeros(1,cubeL*cubeL*cubeL);
parfor i=1:cubeL*cubeL*cubeL
    ourval(i,:)= some_function(xyz(i,:),P_labs,rgb,XYZw,isave,i,cubeL,method,ourval);
    isave(i)=i;    
    send(q, i,ourval,isave); 

end


send(q, 0);

Thank you for your time.

[Nicolas B.]

Hi,

I haven't much time so I haven't tested your code. However, when calling a parfor loop, you must already have created the data at the right size (no resizing in parfor for common data). So try to create ourval before parfor.

For the code 2, try to write send(q, num2str(i)) in the loop instead of what you have. If what I suspect is right, I think that it should call disp(i) at the end of each loop.

I let you try. If I find time this afternoon, I will try to run your code.

Regards