Issue with parfor when using containers.Map

Here is the code of a script that calls a function inside a loop. The function is passed a Map object (created previously). I entered in the comment the details of the Map. The function actually does nothing with the map. I run this code with a for loop and it takes little time of course:

"Elapsed time is 0.000417 seconds."

When I change the for to a parfor and run again - it goes into a black hole. I have to kill it. In the Activity Monitor (on a Mac) I can see all 8 processors are 100% busy.

I assume it may have to do with the distribution of a large map, but I tried it also on a server with 1.5TB of memory and it hangs there too.

Any suggestions or explanantions will be greatly appreciated.

Perhaps there is a better way to distribute the map as a shared variable (it is not modified inside the loop)

Here is the script:

% >> myMap
% 
% myMap = 
% 
%   Map with properties:
% 
%         Count: 10344256
%       KeyType: char
%     ValueType: any
% 
% >>
tic
for i=1:8 % if this for loop is cahanged to parfor - it hangs
    [x,y] = foobar(myMap,i);
end
toc;
function [x,y] = foobar(myMap,i)
x=[];
y=[];
end

[Walter Roberson]

It would be interesting to see what happened if you use parallel.pool.Constant

[Shlomo Geva]

Thanks for the suggestion.

I tried this. A slight improvement, but still a big problem.

I used the parallel.pool.constant to pass the constant instead of myMap. The program does not hang, but it still takes 23 seconds which is several orders of magnitude slower than a for loop.

BTW, I noticed in the activity monitor that each of the workers consumed about 8.2GB of memory (the Matlab workspace itself take about 7.4GB of memory.) So it appears that each worker receives a copy of the map.

[Walter Roberson]

Yes. parfor() and parfeval() do not use shared memory (or at least are not documented as doing so).

In a discussion within the last few days I showed from the R2023a documentation that as of R2023a it cannot be the case that parpool constants are "broadcast" to all workers in a single packet. I see you are using R2022b, at which time such a "broadcast" was at least a hypothetical possibility.

You might want to experiment with using backgroundPool and parfeval as background pools can use shared memory and are designed to reduce the amount of memory copying required.

[Shlomo Geva]

Will do. Thanks for your time.