Quickly finding maximum value of a large (~8,000,000) vector
2 visualizzazioni (ultimi 30 giorni)
Mostra commenti meno recenti
I was trying to see if I could speed up one of my functions so I ran it through profiler. The script loads big files and writes out smaller portions of the data for me to use later. Before writing however, I check to see if the data has any abnormally large values and if it does, I skip it. After looking at the profiler output, I saw that checking to see if there are any abnormally large values is what takes the most time by far (for 70 files, about 450 seconds out of 550 seconds). I never would have thought would have been the slowest part of the code so I'm wondering if I'm missing something that I could easily do to speed it up.
So here's the abridged version of the code:
for i = 1:nFiles % loop over files
Data = load_data(file(i)); % data is a structure with metadata and the actual
% data is in Data.trace which is 8640001x10x3
for s = 1:10 % loop over stations
badStation = 0;
for c = 1:3 % loop over components
% if one component is bad, skip the station
if max(abs(Data.trace(:,s,c))) > badThreshold
badStation = 1;
break;
end
end
if ~badStation
for c = 1:3 % write loop
write_data(Data.trace(:,s,c);
end
end
end
end
I've checked that memory isn't an issue as there's still 9 GB free during run time. Anything simple I can do to improve this?
Thanks in advance.
0 Commenti
Risposta accettata
Più risposte (2)
Walter Roberson
il 22 Gen 2013
You can vectorize a bit:
for a = 1 : 10
if max( reshape( abs(Data.trace(:,s,1:3)), [], 1) <= badThreshold
for c = 1 : 3
write_data(data.trace(:,s,c));
end
end
end
Jan
il 24 Gen 2013
Modificato: Jan
il 24 Gen 2013
See FEX: anyExceed : This can test the values of the array on the fly, such that abs(data) is not created as a temporary variable. The testing stops early, when any element outside the wanted range is found. Therefore a positive test can be found without searching the complete array.
Vedere anche
Categorie
Scopri di più su Data Type Identification in Help Center e File Exchange
Prodotti
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!