Azzera filtri
Azzera filtri

tall array within a parfor?

3 visualizzazioni (ultimi 30 giorni)
Ive J
Ive J il 6 Apr 2022
Modificato: Ive J il 6 Apr 2022
Consider the following scenario:
ds = datastore(...);
dst = tall(ds);
parfor ...
% warning here: dst is a broadcast variable. This might result in unnecessary communication overhead.
chunk = gather(dst(:, idx));
Given the fact that dst is a tall array, why MATLAB warns about overhead? In fact, due to the lazy nature of tall arrays, above parfor is quite memory friendly.
So, what's the reason of the warning in case of tall arrays (pleast note I'm aware of communication overhead issue in case of other data types)?

Risposta accettata

Matt J
Matt J il 6 Apr 2022
Modificato: Matt J il 6 Apr 2022
You've generated idx in a way that doesn't have a simple dependence on the loop variable. This warning is not present if you are simply parfor-looping over the columns.
parfor i=1:size(dst,2)
chunk = gather( dst(:, i) );
Because of whatever you're doing, the variable is not sliced. Because it is not sliced, you may have overhead due to two or more parpool workers trying to access the same memory at the same time. If you had been writing into the tall array, you would have to be even more careful. Parallel write operations into the same memory location can have unpredictable effects.
  1 Commento
Ive J
Ive J il 6 Apr 2022
Modificato: Ive J il 6 Apr 2022
I see, makes perfect sense. idx is a cell array of variable names in dst in non-overlapping chunks.
idx = {["var1", "var2"], ["var3", "var4"], ...};
parfor i = 1:numel(idx)
chunk = gather(dst(:, idx{i}));
For what it's worth, this has (several benchmarks) a stable memory usage (no change during the course of parfor loop), while in case of non-tall arrays out of memory can happen. so, in the end, I feel this is a safe approach regardless of slicing issue

Accedi per commentare.

Più risposte (0)


Scopri di più su Parallel Computing Fundamentals in Help Center e File Exchange




Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by