# Why is the gpuArray version of my code slower?

26 views (last 30 days)
Ariel Lanza on 15 Jul 2019
Commented: Walter Roberson on 17 Jul 2019
Hello,
I am doing some experiments with gpuArrays with a simple problem: for every pair of distinct rows of a random matrix I want to find the difference of the sum of the elements in the rows. (I am aware that this formulation of the problem is linear, however let'assume that the function I am applying to the couple of rows is not linear)
n = 100;
gpu = true;
if gpu
returns = rand(n, 'gpuArray');
ansmat = zeros(n*(n-1)/2, 3,'gpuArray');
else
returns = rand(n);
ansmat = zeros(n*(n-1)/2, 3);
end
itermat = 1;
for v1 = 1:n
for v2 = 1:v1-1
ansmat(itermat,1:2) = [v1,v2];
itermat = itermat + 1;
end
for v2 = v1+1:n
ansmat(itermat,1:2) = [v1,v2];
itermat = itermat + 1;
end
end
tic
for i=1:size(ansmat,1)
sum1 = sum(returns(ansmat(i,1),:),2);
sum2 = sum(returns(ansmat(i,2),:),2);
ansmat(i,3) = sum1-sum2;
end
toc
When I set
gpu = true;
I get
>> ZZZ
Elapsed time is 3.120591 seconds.
When I set
gpu = false;
I get
>> ZZZ
Elapsed time is 0.016989 seconds.
I think I am missing a few fundamentals. For example, when I tried to rewrite my code in terms of arrayfun with
myfun = @(x, y) sum(returns(ansmat(x,1),:),2) - sum(returns(ansmat(y,2),:),2);
tic
ansmat(:,3) = arrayfun(@(x,y) myfun(x,y), ansmat(:,1), ansmat(:,2));
toc
it works without the GPU, however using the GPU there is a problem:
>> ZZZ
Error using ZZZ (line 33)
Use of functional workspace is not supported.
But I could not find those tips.
What is the correct way to gain some speed in solving my problem using the GPU?
Walter Roberson on 15 Jul 2019
Using data as indices is particularly inefficient on GPU.

Stephane Dauvillier on 15 Jul 2019
OK,
Basically if your program takes less than a second on your CPU it's not a surprise it will be slower in a GPU or even in a distant cluster.
I'm suppose you have a convenient GPU.
When dealing with parallel computation (because it's the same thing with "normal" parallel computation) you have 2 set of time: computation and communication. When the communication is bigger than computation time, that's when you have better performance between sequential and parallel.
Let's imagine your CPU as one person and the cluster (in your case GPU) as a team of 10 people.
Let's assume you want the result of the following operation 1+2+..+n. (Let's assume you don't know the formula 1+2+...+n = n*(n+1)/2).
If n is very big then you will use more time that the 10 people (each will compute a 10 th of the computation and then someone will sum the results of each one)
Now, imagine n isn't huge at all (let's say 20). Then you will spend too much time splitting the work the team comparing to the time they will spend to actually do the computation.
Bonus: After doing calculation on GPU retrieve the reuslt on your MATLAB session (meaning get back the result from GPU to CPU with the function gather.
Walter Roberson on 17 Jul 2019
You need to know something about how cuda works. Cuda has a whole bunch of arithmetic units, but it only has one instruction decoder for groups of arithmetic units. When a particular instruction is to apply to some of the arithmetic units (some of the data locations) managed by a particular controller but not others, then that is handled by having the controller send "pause" to the unselected units and the selected units execute the instruction.
Now when you index by a scalar, at most one of the arithmetic units managed by a controller is going to be activated and the rest are going to be paused for the instruction. It is like going through a bunch of student solo recitals one by one, each arithmetic unit getting its time in the spotlight while the others stand around. If you were indexing an array of length 1000 element by element, then each of the arithmetic units would be idle 999/1000 of the time.
Reformulation of code from indexing to vectorized can make a huge difference for cuda.

Andrea Picciau on 16 Jul 2019
Edited: Andrea Picciau on 16 Jul 2019
Hi Ariel,
There are three problems with your script.
• Your code is doing a lot of for loops and indexing of gpuArray data, which is killing performance (this would happen with any GPU code, not just MATLAB!). Have a look at this answer to see why this is is a problem and how you can improve your code.
• Your matrices and vectors are too small. You need matrices of at least 1000x1000 elements to amortise the cost of data transfer to and from the GPU.
• You're not measuring GPU performance correctly. You should use timeit and gputimeit, like I'm discussing in this other answer.

R2019a

### Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!