Detecting missing data, tangent = 0, for given data.

Hi! I have some data given in a file, where some data has gone missing. When plotting the data against time, this missing data appears as a flat plot (tangent = 0) against the respective time. For information, there are about 30 measurement per second. I am interessted in detecting when there is missing data for more than 2 seconds, and if you are feeling brave; displaying for which time or the elements of the missing data (for more than 2 seconds). If there are more than one place where there are missing data for more than 2 sec, they all should be given.

I have tried to approach this by both detecting if the data has not changed from the current index to the next, and when the tangent is about 0.

This is my code per now, but it is not done, nor any good.

Thank you in advance!

%%% With Tangent %%%
data = load('x.mat');
Y = tan(data.XPOS);     %This is the x-positions from the file
crit = find(Y<=0.000001);
arr = [];
 
for i=0:length(crit)
    while crit(i)==crit(i+1)       %checks if the tangent for the following element is also 0
        counter = counter+1;
        if counter > 60     %about 2 seconds
            for j=0:length(counter)     
                arr = crit()    %want to add all the elements which the tangent is 0 (for 2 seconds), don't know how
            end
        else 
            counter = 0;    %if the tangent is 0 for less than 2 sec, reset the counter
        end        
    end
end

Answer 1

[Sergey Kasyanov]

Hello!

The approach is very simple: get array with 1 when Y = 0 and 0 otherwise, then detect when 0 changes to 1 and 1 changes to 0 (by diff), then reshape array with position of changing to convinient form and get difference between position in pairs. If diffrerence is greater than 60, then save it in c and print the message.

c = reshape(find(diff([nan, Y, nan] <= 0.000001)), 2, []);
c(1, :) = c(1, :) + 1;
c = c(:, diff(c) > 60);
for i = 1:size(c, 2)
    fprintf('Times from %i to %i is missed\n', c(1,i), c(2,i));
end

[Vilde Solberg]

Hi! Thank you for the respons.

When I run you code I get 304 time-intervalls for missing data, while in reality there's only 4 intervalls. It prints time intervalls where Y is not 0. I don't know why this is.

[Sergey Kasyanov]

It is maked to counting Y <= 0.000001. You should to change 0.000001 to 0 in first line of code if you want to detect only zeros values.

[Vilde Solberg]

It still gives 306 intervals. It interprets also some of the steapest points for the plot as missed data point.

Again, thank you for helping me :)

[Sergey Kasyanov]

Are there any Y < 0?

[Vilde Solberg]

Yes. I tried to change the first line to '= 0' but I got a error. When using '== 0' and '== 0.00001' I got no errors, but the code did not return/print anything.

[Sergey Kasyanov]

If there are Y < 0 try that first line:

c = reshape(find(diff([nan, abs(Y), nan] <= 1e-6)), 2, []);

[Vilde Solberg]

I'm sorry, it still won't work. I get the right results while using my code, though it's not as user-friendly:

[file,folder]=uigetfile;
filename=fullfile(folder,file);
data = load(filename);      %Reads the data file
mps = length(data.Time)/(data.Time(end));       %measurements per second (around 28)
M = [];
arr = [];
counter = 0;
nc = 0;
nnc=0;
intervall=[];
for i=1:length(data.XPOS)-1       
    if data.XPOS(i) == data.XPOS(i+1)
        counter = counter+1;
        if counter > mps*2    % if element is repeated for more than 2 sec
            nc = nc+1;
            arr(nc) = data.XPOS(i);
            if not(ismember(data.XPOS(i),M))
                nnc=nnc+1;
                M(nnc) = data.XPOS(i); 
                intervall(nnc) = data.Time(i);  %the time of the critically missed data
            end
        end
    else counter = 0; 
    end    
end