Use Parfor to split a 4D matrix into several 3D matrices
5 views (last 30 days)
so I to change my current code:
save_file_name=sprintf('layer_%i.mat',i); %name of the save file
variable_name=sprintf('layer_%i',i); %name of the variable
reduced_dimension=squeeze(reducedData(:,:,:,i)); %reduces the size of the matrix down to 3 since
eval(['layer_' num2str(i) '=reduced_dimension;']);%renames the variable
save(save_file_name,variable_name);%saves the files
eval(['clear ', 'layer_' num2str(i) ])
This code is supposed to split a giant 4D Matrix into several smaller 3D matrices while automatically renaming and saving them separatly. I am aware that using eval is considered to be bad, but so far I have not found any other solution.
I want to speed up the whole process by using a parfor loop, but apparently "eval" and "save" are not allowed in this kind of loop. Has anyone an idea how to solve this kind of problem ?
Titus Edelhofer on 4 Sep 2018
I agree with OCDER: parfor will only do huge memory movements from the Client to the workers but I doubt you will get much speedup.
If the "save" is indeed too slow, you might experiment with saving without compression: use "-v6" as flag when saving. The files are then probably much bigger but often saving is faster.
And by the way: if you have the file name with the layer number, why bothering what the variable name is? Why don't simply use "layer" for all of them?
Stephen23 on 4 Sep 2018
Edited: Stephen23 on 4 Sep 2018
Firstly it is important to note that processing a sequence of .mat files is much simpler when the variable names inside each file are exactly the same. This applies to both reading and writing them. Then you could trivially do this:
for slice = 1:d(4)
file = sprintf('layer_%d.mat',slice);
data = squeeze(reducedData(:,:,:,slice));
Note how I wrote simpler, much more efficient code than what you have in your question, and that I saved the slice value too, so you know exactly which slice it is when you load that data:
S = load(...)
Simpler to write, simpler to read, much more efficient: why waste your time fighting eval ?
If you really want to have lots of numbered variables (which are a sign of bad data design, and bad code), then you could split the data into the fields of a structure, and save that structure using the '-struct' option. But of course that would be slow, complex, and not recommended.