Matlab script is very very slow! Help me please!

Hello! I am a student and I created a Matlab code that allows me to open many txt files in succession and perform these sums and other operations. The work that the code has to do is a lot but it is really too slow. I calculated that it would take two days to finish. I think there is some conflict. Please ask me to help me otherwise I will have to do all those operations by hand and it will be a disaster for me. Thanks to those who will help me

I enclose the code I wrote in a text file

[Image Analyst]

Can you also attach 2 or 3 of your text data files so we can at least run the code ourselves? Make it easy for us to help you, not hard.

Tutorial-how-to-ask-a-question-on-answers-and-get-a-fast-answer

[Rocco De Luca]

You're absolutely right, sorry. I put a mega link where you can download the files. I can't attach them here because they are more than 7 mega and it doesn't accept rar files. Thanks so much.

https://mega.nz/#F!DlsCnAoL!H4kt4t5QX-5-ACj8rr09Tw

[Rocco De Luca]

These are hourly powers related to a simulated hybrid system.

Answer 1

[per isakson]

Why not just let it run over a weekend?

See Techniques to Improve Performance and Profile to Improve Performance

I've downloaded your files and invoked your script. I interrupted the execution during the fourth data file. The script consists of three nested loops. One statement in the innermost loop uses 90% of the execution time.

for ii = 1 : 512
    ...
    for kk = 1 : 8760 % What about leap years? 
        ...
        for jj = 1 : 8760 % What about leap years? 
            R(ii,[1:5,8:27])=R(ii,[1:5,8:27])+intermedia(jj,[1:5,8,9,11:21,29:35]); % 90%
            ...
        end
    end
end

The innermost loop does a number of summations. Replacing the innermost loop by

R(ii,[1:5,8:27]) = sum( intermedia(1:8760,[1:5,8,9,11:21,29:35]), 1 );
is_positive = intermedia(1:8760,7)>0;
R(ii,6) = sum( intermedia(is_positive,7) );
R(ii,7) = sum( intermedia(not(is_positive),7) );

improves the speed of the script significantly. (You need to confirm that it returns the correct result.)

[Rocco De Luca]

Thank you infinitely! I ran the code with your replacement, running it for the first file "COMPLETOA20KM1" (so without the most external for) the results seem correct and the execution time is about 25.5 seconds, so for all 512 files it will take 3 and a half hours a lot better than the 48 before! So the problem is nested for loops. Thank you again, if there are other tricks to speed up this code I would like to know them!

[per isakson]

Did you try the profiler?

I don't think there are more low hanging fruits.

[Rocco De Luca]

Yes, I used the profiler. I attach the profiler and the code that I used since I slightly modified it.

Answer 2

[Rocco De Luca]

Hello and sorry if I still disturb you. To speed up even more I thought of replacing the second for with a method similar to the one you used. I am attaching the code, what do you think?

[per isakson]

Assume that your original code with the three loops produce the expected result. Use that result as a reference. Each new version shall produce exactly (a floating point error is ok) the same result. That can be tested systematically with the Unit Testing Framework. If the results are identical and if there is enough variation in the data files, COMPLETOA20KM*, the new code is most likely correct. Are the results identical?

[Rocco De Luca]

Thanks, the results are identical, but now the code is extremely faster. I thank you very much for your help.