Efficient Moving Quantile Function

Hi,

I am looking for thoughts on a recipe to perform a moving quantile in Matlab. Essentially I am trying to obtain the 98% value, for example, of a window that is instantaneously tracked through a large data set. I have tried looping in the quantile function, but this is extremely slow. I have had reasonable success using the sort function each time the window is moved, and then selecting the value based on the index that is 98% of the window size - see script below. But it is still several orders of magnitude slower than the built-in movmedian function. I don't see how a moving quantile should be much different then a moving median, just that the sorted value selected is not at 50%.

My thoughts are that after the initial window is sorted, an insertion sort algorithm is most appropriate since only a single value comes into the window each time the window moves 1 point. But really I am guessing at what is inside the movmedian function. I am not a programmer guy, so any advice here or examples on how to code an efficient moving quantile function is appreciated.

The example below is relatively quick with the given sampling rate, however in my example the sampling rate is 100,000 sps and takes much longer.

    % Moving Quantile
    fs=1000;                        % sampling rate (sps)
    timeData=0:1/fs:40;             % time (sec)
    Data=rand(1,length(timeData));  % data (EU)
    window=6300;                    % number of points used in 1 pt sliding window (pts)
    dt_wnd=(window-1)/fs;           % window time span (sec)
    time_wnd=timeData(1)+dt_wnd/2:1/fs:timeData(end)-dt_wnd/2;    % time vector, sliding window center points (sec)
    
    Ind=floor(0.98*window);                         % index at 98% quantile
    for j=1:length(time_wnd)                        % loop through all windows moving by 1 pt
            SortedData=sort(Data(j:j+window-1));    % sorts data in the jth window
            Val(j)=SortedData(Ind);                 % value at index
    end

    
    % Plotting
    plot(timeData, Data)
    axis([0 40 0.9 1.0])
    hold on
        plot(time_wnd,Val,'LineWidth',4)
    hold off
    title('Moving Quantile')
    xlabel('time, s')
    ylabel('Amplitude')

[Benjamin Stucky]

I am also interested in moving quantiles. The implementation of @Eike Petersen works great for smaller data sets. But I am dealing with very large time series. I found this C# reference implementation: https://aakinshin.net/posts/partitioning-heaps-quantile-estimator/ As I have no knowledge about C# and C++ wrappers, I am not able to create a mex file from this. If somebody could help to implement it, this would be nice.

Answer 1

[Eike Petersen]

My guess is that your guess is pretty good. :-)

I just hand-implemented a version of this with a naive insertion sort algorithm, and it's roughly 50-80x slower than medfilt1. I'm not sure you can get much quicker without switching to C / a mex-file.

You can find it here; it works for N-D arrays, with NaNs, and with different padding strategies: https://de.mathworks.com/matlabcentral/fileexchange/84200-movquant