How not to copy constant large data for every worker in spmd.

Hello everyone.

I am using a spmd statement as follows

spmd
    n1 = 1+sizeClusters*(labindex-1);
    n2 = sizeClusters*labindex;
                
    my_output = my_function(Data1(:,n1:n2),Data2);
end

In this statement Data2 happens to be a very large matrix that is copied for each worker resulting in memory issues. Moreover, my_function needs to read all the values in Data2, therefore I cannot split it in portions like I do for Data1.

Such large matrix is constant and never modified into my_function, therefore such copying for each worker is totally unnecessary.

Is there a way to avoid to copy this large matrix and make the workers to (only) read from the same data?

[Matt J]

The best strategy probably depends on what my_function is doing with Data2. Even if it eventually incorporates all of Data2 into whatever calculation it is doing, there surely has to be some way to break down the calculation into some sequential steps - there is no such thing as a digital computer that reads all data from a large array in perfect simultaneity...

[Edric Ellis]

As Matt says, without any details of the operation of my_function, it's hard to know what's the best strategy here. For example, it might be possible to distribute the contents of Data2, and transfer the bits of Data1 as necessary.

Answer 1

[Matt J]

Instead of reading Data2 into the workspace, you could keep it in a .mat file, and have each worker access it as needed through a matfile object.

https://www.mathworks.com/help/matlab/ref/matlab.io.matfile.html