race condition at asynchronous task assignment during runtime
6 views (last 30 days)
Dear Matlab community,
I am trying to figure out a problem that I anticipated easy but is proving me wrong ... Basically, I am trying to solve a race condition that arises using the parallel computing toolbox.
Imagine I have N tasks and G GPUs. Each task will take a different amount of time to be completed, and this time can be determined only during runtime. Thus, I don't want to preassign the N/G tasks to each GPU, as this will lead to an unbalanced workload distribution. Instead, I wish to launch an spmd of G labs (each one controlling a GPU) so that , whenever a lab finishes one assigned task, a new one is assigned to it in runtime, till all N tasks have been finished.
The problem with this concept is that each lab needs to know which tasks have already been assigned to other labs.
In one of the approaches I've tested, I create a file that stores the last task number. The lab reads this file, increases the task number by one and updates the file. Here the code (for N=3 tasks and G=2 )
numberTasks = 3;
lastAssignedTask = 0;
file = 'lastAssignedTask.txt';
lastAssignedTask = dlmread(file);
fprintf('Lab:%d read that the last assigned task was %d \n',labindex,lastAssignedTask);
taskForThisLab = lastAssignedTask+1;
lastAssignedTask = taskForThisLab;
fprintf('task: %d Lab:%d \n',lastAssignedTask,labindex);
However, the output is
Lab:1 read that the last assigned task was 0
task: 1 Lab:1
Lab:1 read that the last assigned task was 1
task: 2 Lab:1
Lab:1 read that the last assigned task was 2
task: 3 Lab:1
Lab:2 read that the last assigned task was 0
task: 1 Lab:2
Lab:2 read that the last assigned task was 1
task: 2 Lab:2
Lab:2 read that the last assigned task was 2
task: 3 Lab:2
Looks like each worker reads the file at the same time. I have been trying several workarounds (see the tags,) but none seems to be useful for this problem. Is there somehitng very obvious that I am overlooking?
Walter Roberson on 30 Dec 2020
To prevent race conditions, instead of having each lab work independently, you can use an extra lab as a controller that manages the task assignments. The labs wait for work with a labReceive(), and notify the controller that they are done with a labSend() . The controller assigns work to any device that does not have work, and then does a labReceive() waiting for response. When there is no more work, the controller signals shutdown.