Quickly finding maximum value of a large (~8,000,000) vector

I was trying to see if I could speed up one of my functions so I ran it through profiler. The script loads big files and writes out smaller portions of the data for me to use later. Before writing however, I check to see if the data has any abnormally large values and if it does, I skip it. After looking at the profiler output, I saw that checking to see if there are any abnormally large values is what takes the most time by far (for 70 files, about 450 seconds out of 550 seconds). I never would have thought would have been the slowest part of the code so I'm wondering if I'm missing something that I could easily do to speed it up.

So here's the abridged version of the code:

for i = 1:nFiles % loop over files
  Data = load_data(file(i)); % data is a structure with metadata and the actual
                             % data is in Data.trace which is 8640001x10x3
   for s = 1:10 % loop over stations
      badStation = 0;
      for c = 1:3 % loop over components
         % if one component is bad, skip the station
         if max(abs(Data.trace(:,s,c))) > badThreshold
            badStation = 1;
            break;
         end
      end
      if ~badStation
         for c = 1:3 % write loop
            write_data(Data.trace(:,s,c);
         end
      end
   end
end

I've checked that memory isn't an issue as there's still 9 GB free during run time. Anything simple I can do to improve this?

Thanks in advance.

Answer 1

[José-Luis]

http://www.mathworks.com/matlabcentral/fileexchange/23576-minmax-selection

[William]

This did the trick! Thanks.

Answer 2

[Walter Roberson]

You can vectorize a bit:

for a = 1 : 10
  if max( reshape( abs(Data.trace(:,s,1:3)), [], 1) <= badThreshold
    for c = 1 : 3
      write_data(data.trace(:,s,c));
    end
  end
end

[William]

I just tried this out and it runs slightly faster (1-3%), but it's definitely cleaner!

Thanks.

Answer 3

[Jan]

See FEX: anyExceed : This can test the values of the array on the fly, such that abs(data) is not created as a temporary variable. The testing stops early, when any element outside the wanted range is found. Therefore a positive test can be found without searching the complete array.

[William]

Wow, this is a pretty cool function and is also quite helpful. Thanks.