How to find the inflections points in noisy, variable data cycles?

My data looks like the attached photo and I am trying to find the circled points. I have tried findpeaks but it does not return everything and frequently makes mistakes. This is due to not being able to consistently set a minpeakdistance, or minpeakheight. I have also tried the findchangepts function to do a slope average and find the intersections but with the noise it also fails every time. What is the best approach for doing this?

These points are defined by a change in overall slope in the data (since there is noise in the data set). I want to find the last data point before the slope sharply plunges, and the last data point before it increases sharply again. The circles are just to indicate where those points of inflection generally fall in the data (yes I am aware they are not precise). As far as I know findpeaks does not allow for slope definition, i.e. find the data point where the slope of the 10data points to the left is below X, and where the 10 data points to the right is above Y, or something to that effect.

[Jan]

The circles have a certain diameter and it is not clear, which points you want to select exactly. Start with defining mathematically, how these points are defined. What about the last edge? Which point would you select manually?

So it is not findpeaks, which makes "mistakes", but it works as it is is instructed. The problem is not trivial and in some cases, the decision based on the mathematical definition might differ from the points you select by intuition.

[no nobody]

These points are defined by a change in overall slope in the data (since there is noise in the data set). I want to find the last data point before the slope sharply plunges, and the last data point before it increases sharply again. The circles are just to indicate where those points of inflection generally fall in the data (yes I am aware they are not precise). As far as I know findpeaks does not allow for slope definition, i.e. find the data point where the slope of the 10data points to the left is below X, and where the 10 data points to the right is above Y, or something to that effect.

[Jan]

If you want to consider the slopes, what about appling gradient at first and search for the peaks there? The noise remains a severe problem. Hm.

[Greg Dionne]

You could also try getting the indices by using midcross. Then backtrack to the local maximum. Your signal looks fairly noisy, but you could also try using overshoot and undershoot using a time-reversed signal.

Answer 1

[John D'Errico]

I think you are looking at this the wrong way.

You are looking to use findpeaks. But really, you should look for where in the sequence the gradient is very large (positive or negative.)

Then step backwards just a bit, and find the peak which just precedes the high gradient region.

The issue is, your data is noisy as hell. There are large spikes in the high gradient region where the sequence completely reverses itself.

Remember that what seems easy for your eyes/brain to see, is often far more difficult to find by computer. You can decide that some stuff is insignificant, and can be ignored.

One option might be to try training a neural net to identify the peaks of interest. I'd bet if you could get a sufficient number of events, a neural net would work nicely. But then I'm not an expert on them.

[no nobody]

My signal is noisy, but it is also fairly regular, so I was thinking that maybe if I did like a 20point average of slope, and then checked where it changes I could get pretty close to the points I want +/- hopefully 3-5pts. (this is sampled at 50Hz).

Would this work? I have no idea how to implement the running slope average though.

[John D'Errico]

To get a 20 point moving slope estimate, you need to use a tool like my movingslope tool. It is on the File Exchange.

it computes an estimate of the slope of the series at every point, using a running least squares fit along the curve. You can define the order of the local polynomial fit as well as the width of the moving window that will be employed, all using a Savitsky-Golay scheme.

If instead, you are just asking how to compute the local point-to-point slope at each point, and then compute a running mean of the slope, then diff will compute the local slope between each pair of points, then use a trick like

conv(S,ones(1,20)/20,'valid')

to compute a running windowed mean of those slopes. You would then patch the ends to extend the moving mean of slopes to each end.

Note that diff is a noise amplification process, so you might arguably be better off using the movingslope utility to try to reduce the noise in that estimation.

Would it work? In any case you still need to be able to find that point. This is why I suggested a neural net approach. Given several such time series, locate the points you want it to find, doing this part by hand/eye. Then train the neural net to locate the indicated points. Now you have a tool that knows how to identify exactly what you saw when you did the work by eye. The point is, neural nets are really good at pattern recognition, and that is what you want to do here.

After some additional thought, I think you could use PCA to do a similar task. Neural nets would do the job better, IMO.

Answer 2

[Jan]

If the slopes are mot important, use gradient(signal) as input for findpeaks.

[no nobody]

I tried that but it does not catch all of the peaks/troughs, and several of them are off by a bit. What would this look like in terms of the code? something like:

 [Maxima,MaxIdx] = findpeaks(Peakdata,'MinPeakHeight',5,'MinPeakDistance',50,'MinPeakProminence',10,'Annotate', 'extents');

Peakdata is the gradient of my data. This misses quite a few, I can tweak the distance and prominence, but never perfectly (basically guess and check) to get all the points or even 75+%

[no nobody]

So here is a zoom of some of my raw data. As you can see there is usually a cluster of points (small changes in slope mostly due to noise), and then a sharp change with few points, followed by another cluster, and then a sharp rise. This is the only consistent feature of my data, how can I exploit this to find the points I want? What would something look like where I could say find the point where the next 3 points have a slope (linear fit of some kind) have a negative value of at least X, and then the reverse for the bottom value I am trying to find?

[Greg Dionne]

Could you post some of your data?

[Jan]

@no nobody: I still see the same problem: Which points do you consider as correctly chosen? You did not provide a mathematical criterion.

 checked where it changes I could get pretty close to the
 points I want +/- hopefully 3-5pts

Which are the "points you want"? Maybe you mean the peaks in the 2nd derivative? Please mark these points manually in the screenshot, because this is better than letting the readers guess, what you exactly want.