Matlab clears persistent and global variables in fmincon when activating UseParallel

It seems matlab clears all global and persistent variables within the objective function when using "UseParallel" in fmincon. I am trying to track each function call within the objective function (this is because my objective function should generate a specific input file to be called by an exteranl program and give a misfit output).

I use the folowing code to within my objective function to generate the unique file name at each iteration:

persistent count_1
if isempty(count_1)
    count=0;
else
    t = getCurrentTask();
    worker_number = t.ID;
    c = worker_number * 10000 + count_1;
    count =c;
    
    count_1=count_1+1;
end
count_1=count_1+1;

filename = ['Run_' num2str(cout) '.dat'];

But strangly, the name would not change after the first function call. Have you encoutered this issue?

Answer 1

[Matt J]

This happens because, in parallel computing, it isn't easily definable what it means for a variable to "persist". A peristent variable is normally one that retains its value from the previous function call. But when the objective function calls are all executing simultaneously (or at least non-sequentially), what does "previous" even mean?

With global variables, similar ambiguities arise. What if two parallel-occuring executions of the objective function assign different values to a global variable G that they share with the base workspace? Which of the conflicting values should reside in the base workspace version of G after the execution is complete?

[H R]

I see your point @Matt J. I just wanted to generate sequential file names. It seems fmincon first calculate f(x0), with x0 being the starting point (without the parallel option). Then since I do not know the gradients, it takes n function evaluations (n is the number of variables) by perturbing each variable at the time (this is done in parallel) to generate the gradients using the finite difference. Then a few Newtonian iterations. This process is repeated multiple times. I would not care about the finite differencing names, but it would be great to generate sequential names for the Newtonian iterations.

[Matt J]

What about line search operations? Would you want output files for the function evaluations where fmincon simply tests potential points along a search line?

In any case, perhaps you can have,

optimoptions('fmincon','SpecifyObjectiveGradient',true,'UseParallel',false); 

but supply your own finite differencing gradient calculations within your objective function code. To have the finite differencing parallelized, you can invoke parfor or parFeval yourself within this function (this is all UseParallel really does). Setting it up this way will give you full control over if and when output files are generated.

[H R]

I think this is a good option to try, thank you

[Matt J]

Another option might be to generate output files in an OutputFcn,

https://www.mathworks.com/help/optim/ug/output-functions.html#OutputFunctionsForOptimizationToolboxExample-2

This lets you detect which particular sub-stage of the iteration the code is currently in, and generate some sort of customized output accordingly.