Removing jumps from data when plotting a graph

I am trying to plot sensor readings. See attached the data and the graph. Please I need a help on how to remove those jumps in the graph (space between the 2 red markings). I would like to apply same to the remaing jumps in the graph. Any help is well appreciated.

[Matt Gaidica]

Austin, need some more details (what kind of sensor is this?). For example, if I zoom in on one of the smaller discontinuities, it's not a very sharp jump, it kind of stumbles downwards.

• What information are you trying to maintain?
• What constitutes a "jump" (i.e. dy/dx)?

Have you tried plotting diff() of your signal, and does that highlight the trouble areas sufficiently?

[Austin Ukpebor]

Thanks Matt for your response. I tried to plot the diff() of my signal but did not produce what I want.

It's a hall-effect sensor used with a combination of magnet to measure the gaping (opening and closing) of oyster shells. I want to retain those thick areas of the graph. That thick areas tell me about the spawning of the animal.

[Star Strider]

If you are calculating the numerical derivative, use the gradient function. It produces an output that has the same dimensions as the input.

[Austin Ukpebor]

Thanks Star Strider. I couldn't figure out on how to apply gradient function to my case using the link you shared. Please help me out, I have my data attached.

@Matt Gaidica, please share the code you used to generate your graph. When I used the diff() function, I had a different graph.

Answer 1

[Star Strider]

I am not certain what result you want.

Try this:

D1 = readmatrix('SensorValues.xlsx');

s = D1;                                                             % Signal Vector
L = size(s,1);                                                      % Data Length
Fs = 1;                                                             % <— Need Actual Sampling Frequency Here!
Ts = 1/Fs;                                                          % Sampling Interval
Fn = Fs/2;                                                          % Nyquist Frequency
t = linspace(0, L, L)*Ts;                                           % Time Vector


sc = s - mean(s);                                                   % Subtract Mean (Makes Other Peaks More Prominent)
FTs = fft(sc)/L;                                                    % Normalised Fourier Transform
Fv = linspace(0, 1, fix(L/2)+1)*Fn;                                 % Frequency Vector
Iv = 1:numel(Fv);                                                   % Index Vector (One-Sided Fourier Transform)

figure
plot(Fv, abs(FTs(Iv))*2)
grid
xlim([0  0.001])
title('Fourier Transform')
xlabel('Frequency (Hz)')
ylabel('Amplitude')

Wp = [0.001 0.075]/Fn;                                              % Passband Frequency (Normalised)
Ws = [0.9 1.1].*Wp;                                                 % Stopband Frequency (Normalised)
Rp =  1;                                                            % Passband Ripple
Rs = 60;                                                            % Passband Ripple (Attenuation)
[n,Wp] = ellipord(Wp,Ws,Rp,Rs);                                     % Elliptic Order Calculation
[z,p,k] = ellip(n,Rp,Rs,Wp);                                        % Elliptic Filter Design: Zero-Pole-Gain 
[sos,g] = zp2sos(z,p,k);                                            % Second-Order Section For Stability

figure
freqz(sos, 2^20, Fs)                                                % Filter Bode Plot
set(subplot(2,1,1), 'XLim',Wp*Fn.*[0.8 1.2])                        % Optional
set(subplot(2,1,2), 'XLim',Wp*Fn.*[0.8 1.2])                        % Optional

s_filtered = filtfilt(sos, g, s);                                   % Filter With IIR Filter

figure
plot(t, s)
hold on
plot(t, s_filtered)
hold off
grid
xlabel('Time (Units Estimated)')
ylabel('AMplitude (Units Not Specified)')
legend('Original Signal', 'Bandpass-Filtered Signal', 'Location','W')

producing:

The filter eliminates the d-c (constant) offset, and a bit of the high-frequency noise. Adjust the upper limit of the ‘Wp’ vector to get the result you want.

Note — With the actual sampling frequency, it will be necessary to adjust the limits of ‘Wp’ and the xlim values of the Fourier Transform plot.

[Austin Ukpebor]

Thanks Star for this great contribution. Yes, I am already tweaking Wp to get my desired result. Which of the smoothing techniques would you suggest to remove those 4 unwanted transients/spikes in the filtered signal? Median filtering seems to be a natural way to eliminate them but not working for me. I used this one line code below.

plot(medfilt1(s_filtered +0.2,100))

[Star Strider]

Spikes and other discontinuities can be difficult to deal with.

One option:

D1 = readmatrix('SensorValues.xlsx');

s = D1;                                                             % Signal Vector
L = size(s,1);                                                      % Data Length
Fs = 1;                                                             % <— Need Actual Sampling Frequency Here!
Ts = 1/Fs;                                                          % Sampling Interval
Fn = Fs/2;                                                          % Nyquist Frequency
t = linspace(0, L, L)*Ts;                                           % Time Vector


sc = s - mean(s);                                                   % Subtract Mean (Makes Other Peaks More Prominent)
FTs = fft(sc)/L;                                                    % Normalised Fourier Transform
Fv = linspace(0, 1, fix(L/2)+1)*Fn;                                 % Frequency Vector
Iv = 1:numel(Fv);                                                   % Index Vector (One-Sided Fourier Transform)

figure
plot(Fv, abs(FTs(Iv))*2)
grid
xlim([0  0.001])
title('Fourier Transform')
xlabel('Frequency (Hz)')
ylabel('Amplitude')

Wp = [1E-8   1.5E-3]/Fn;                                            % Passband Frequency (Normalised)
Ws = [0.9 1.1].*Wp;                                                 % Stopband Frequency (Normalised)
Rp =  1;                                                            % Passband Ripple
Rs = 60;                                                            % Passband Ripple (Attenuation)
[n,Wp] = ellipord(Wp,Ws,Rp,Rs);                                     % Elliptic Order Calculation
[z,p,k] = ellip(n,Rp,Rs,Wp);                                        % Elliptic Filter Design: Zero-Pole-Gain 
[sos,g] = zp2sos(z,p,k);                                            % Second-Order Section For Stability

figure
freqz(sos, 2^20, Fs)                                                % Filter Bode Plot
set(subplot(2,1,1), 'XLim',Wp*Fn.*[0.8 1.2])                        % Optional
set(subplot(2,1,2), 'XLim',Wp*Fn.*[0.8 1.2])                        % Optional

s_filtered = filtfilt(sos, g, s);                                   % Filter With IIR Filter
s_filtered(s_filtered < -0.5) = nan;                                % Region = NaN
s_filtered = fillmissing(s_filtered, 'linear');                     % Linearly Interpolate

figure
plot(t, s)
hold on
plot(t, s_filtered, '-r', 'LineWidth',1.5)
hold off
grid
xlabel('Time (Units Estimated)')
ylabel('AMplitude (Units Not Specified)')
legend('Original Signal', 'Bandpass-Filtered Signal', 'Location','W')

producing:

The data in the discontinuity region does not appear to be valid, considering the other parts of the signal, so simply eliminating it may be a worthwhile option.

.

[Austin Ukpebor]

At this point, I am confident I can apply your contributions to tidy up my work! Thanks so much!

[Star Strider]

As always, my pleasure!

Answer 2

[Fangjun Jiang]

In your case, since you have enough data, I think you can use symbol in plot() to remove the "jump".

If your old way is plot(x,y), then you can use plot(x,y,'.')

[Austin Ukpebor]

Thanks Fangjun.

Answer 3

[Matt Gaidica]

What do you want to do with the data that around the jumps (still referring to my previous figure)? Interpolate it? Maybe you want some type of dynamic detrending, but it still doesn't totally remove the artifact of the jumps. I just loaded your data in A.

y = smoothdata(A,'movmean',100);
close all
figure;
subplot(211);
plot(A);
hold on;
plot(y);

subplot(212);
plot(A-y);

[Austin Ukpebor]

@Matt Gaidica, thanks for your continued support. I just observe my expectation in my initial request wasn't well defined. I wanted to retain those data around the jumps while I make the data above or below the jumps to align to preceeding data. See figure 1 below. To answer your query, interpolation is more like it.

I tried modified median signal (see figure 2 below), only median absolute deviation (MAD) is near my expectation but my data of interest was over stretched. Again, applying moving average to the MAD did not provide the required significant changes, see fig 3.

Any approach to get my desired graph is appreciated!

[Matt Gaidica]

Would it be possible to hand select some of those discontinuities as a semi-automatic approach? Given the first figure I showed, you're going to have some issues stitching these data back together.

I think you're going to have to (1) figure out how to detrend or re-align the data then (2) filter remaining discontinuities or smooth it. Here's one last idea, maybe you can get those numbers (e.g., 200, 50, 20) to play nice with your data, or get close enough to analyze it:

y = movmax(A,200);
B = abs(A - y);
TF = smoothdata(B,'movmed',50);
C = B;
C(TF > median(B) * 20) = 0;

close all
ax = [];
figure;
ax(1) = subplot(211);
plot(A);
ax(2) = subplot(212);
plot(B);
hold on;
plot(C);

linkaxes(ax,'x');

[Austin Ukpebor]

@Matt Gaidica, I think we are nearer to my desired expectation. Your last graph looks more like it though the output signal is weigh different from the original signal. What technigue did you apply and please share your code with me.

Again, thanks.

[Austin Ukpebor]

@Matt Gaidica, please I am checking back if you could share the code you used to generate your last graph. I tried diff() function but got different graph. Once again, thanks.

[Matt Gaidica]

I posted the code above. In https://www.mathworks.com/matlabcentral/answers/720594-removing-jumps-from-data-when-plotting-a-graph#comment_1273664

[Austin Ukpebor]

Your code gave me this first graph.

How did you get this figure below? Did you apply smooth function? Thanks

[Matt Gaidica]

That's just zoomed in. Sorry for the confusion.

[Austin Ukpebor]

Matt, thanks for your great contributions. I can now tidy up my work. Appreciated!

[Matt Gaidica]

Sure thing, sounds like a cool project. Feel free to reach out directly if you need anything!