Why does runtime decrease the more times I execute a command?

For example:

>> tic;1*1;toc
Elapsed time is 0.021571 seconds.
>> tic;1*1;toc
Elapsed time is 0.000074 seconds.
>> tic;1*1;toc
Elapsed time is 0.000103 seconds.
>> tic;1*1;toc
Elapsed time is 0.000114 seconds.
>> tic;1*1;toc
Elapsed time is 0.008142 seconds.
>> tic;1*1;toc
Elapsed time is 0.000046 seconds.
>> tic;1*1;toc
Elapsed time is 0.000007 seconds.
>> tic;1*1;toc
Elapsed time is 0.000007 seconds.
>> tic;1*1;toc
Elapsed time is 0.000009 seconds.
>> tic;1*1;toc
Elapsed time is 0.000006 seconds.
>> tic;1*1;toc
Elapsed time is 0.000007 seconds.
>> tic;1*1;toc
Elapsed time is 0.000008 seconds.

Answer 1

[Walter Roberson]

Some of the answer is the Just In Time compiler. Newer versions of MATLAB have added more and more JIT capabilities to the command line (as opposed to function files.)

However, I have multiple times found that the second call to essentially the same code can be much faster than the first, even if the first involves multiple calls to a timeit of a function -- that is, things that should be JIT'd the first call (or at most two.) Along the lines of

N = 1000;
T1A = zeros(1,N);
T1B = zeros(1,N);
T2 = zeros(1,N);
for K = 1 : N T1A(K) = timeit(@() MyFunction1(), 0); end
for K = 1 : N; T2(K) = timeit(@() MyFunction2(), 0); end
for K = 1 : N; T1B(K) = timeit(@() MyFunction1(), 0); end

nv = 1 : N;
plot(nv, T1A, 'b', nv, T1B, 'g', nv, T2, 'k')

then with MyFunction2 being a different function than MyFunction1 it is quite understandable that it would take a different time, but the T1B series of calls to MyFunction1 will be much faster than the T1A series -- and if you rearrange to

for K = 1 : N; T2(K) = timeit(@() MyFunction2(), 0); end
for K = 1 : N T1A(K) = timeit(@() MyFunction1(), 0); end
for K = 1 : N; T1B(K) = timeit(@() MyFunction1(), 0); end

then it is the MyFunction2 T2 series that will suddenly slow down and T1A and T1B will be pretty much the same. The first in the group like this always comes out slower. If it was JIT effects then you might expect the first couple of iterations, T1A(1:5) for example to potentially be slower than T1A(6:end), as the JIT fully kicks in, but in my experience it is never less than about 400 iterations before the times settle down and it is not uncommon for all of the first 1000 to be notably slower than any subsequent series.

[Liyuan]

Do you get graphs like these (if you just focus on the baseline), I don't seem to get the graphs you seem to be getting..

Here's the code I am running, I'm on a mac btw

figure
N = 500;
T1A = zeros(1,N);
T1B = zeros(1,N);
T2 = zeros(1,N);
for K = 1 : N; T1A(K) = timeit(@() magic(20)*magic(20), 0); end
for K = 1 : N; T2(K) = timeit(@() mean(1:100000)); end
for K = 1 : N; T1B(K) = timeit(@() magic(20)*magic(20), 0); end
nv = 1 : N;
plot(nv, T1A, 'b', nv, T1B, 'g', nv, T2, 'k')
ylim([0,0.001])

=

figure
N = 500;
T1A = zeros(1,N);
T1B = zeros(1,N);
T2 = zeros(1,N);
for K = 1 : N; T2(K) = timeit(@() mean(1:100000)); end
for K = 1 : N; T1A(K) = timeit(@() magic(20)*magic(20), 0); end
for K = 1 : N; T1B(K) = timeit(@() magic(20)*magic(20), 0); end
nv = 1 : N;
plot(nv, T1A, 'b', nv, T1B, 'g', nv, T2, 'k')
ylim([0,0.001])

[Walter Roberson]

I gave an example in https://www.mathworks.com/matlabcentral/answers/324130-is-arrayfun-faster-much-more-than-for-loop#comment_428086 and in https://www.mathworks.com/matlabcentral/answers/367479-efficiency-using-struct-field-vs-variable#answer_291590

You might also be interested in my code at https://www.mathworks.com/matlabcentral/answers/348746-is-it-better-to-have-one-function-or-multiple-instances-of-one-code#comment_470713

Answer 2

[Jan]

The observed effect is random. To get real timings use either timeit or at least a loop with a large number of calls to reduce the random effects of flushing buffers, virus scanners, throttling of the processor, cores/RAM/cacheoccupied by other applications, disk access and so on.

timeit(@() 1*1)

Result:

 Warning: The measured time for F may be inaccurate because it is running too fast.
 Try measuring something that takes longer. 
 > In timeit (line 158) 
 ans =
       3.6328987854502e-06

The result of about 4e-6 is more or less reproducible.

tic
for k = 1:1e6
  b = 1*1;
end
toc

 Elapsed time is 0.056205 seconds.

But it is not clear, what this result means. Maybe the JIT acceleration removes the repeated code and the loop might take more time than the processing.

Conclusion: Measuring the time of tiny calculations is not meaningful.

[Liyuan]

But I think the effect is not random, everytime I try to time some operation, it's always the case that the first few times it takes longer, and subsequently the timing stabilizes at a smaller value. In both of your examples this effect is not present because you are taking medians and means (I'm assuming that's what you mean to demonstrate in the second example), is this the effect of the JIT compiler? Is it that because I'm running the same code a few times, the compiler realizes that, and to optimize performance it does something, which makes my code run faster?