finding turning points of a dataset

Dear all, I hope somebody can help me with the following problem: I have a vector of x. The numbers within this vector (1-n) change at a very slow rate (the difference between data points is too small). So you can not say when numbers really increase or decrease. However I can see that at some points the rate of change is getting faster which is in my case a real turning point. How can I check when changes between data points start accelerating? I hope I am clear enough. I really appreciate your help. Sobhan

[Image Analyst]

Please upload a plot diagram of your data somewhere so we can see what it looks like.

[Sobhan]

Hi,

You can see a plot of my data here:

http://www.flickr.com/photos/79773081@N03/7141234993/in/photostream/

Cheers,

Sobhan

Answer 1

[Geoff]

You can use diff or gradient.

Decide what minimum rate of change is acceptable:

tolerance = 1e-4;

And then to find the point of interest:

index = find( abs(diff(x)) > tolerance )

However, this is going to find ALL points that exceed your tolerance. I don't know what your data is, but if you say it accelerates, then every point after the turning point is going to be returned. Also, unless there is a theoretical reason behind your 'small changes', you might need to detect the tolerance. That's always more fiddly.

Another way to go about this is to detect a 'baseline' and remove it from your data. It might not be relevant to you... I've used this to analyse qPCR experiments for gene research. We had sigmoid-like curves where the initial part of the curve was linear. I had to detect that linear section, find a regression line through it, and subtract that line from the data. It might not be relevant to you. The theory behind this is to remove from the data any constant change that is proven to exist and must be corrected.

How many turning points do you expect in each vector, or is this unknown? What does the data look like?

[Vahid]

Hello, I was wondering if you could elaborate the solution you provided a bit more. I'm particularly more interested in the method you used in your qPCR experiments. I have to find the knee point of a sigmoid speech perception function in matlab and trying to evaluate different methods. Thank you very much in advance!

[Ulises Sosa]

Hello,

I have a similar situation. I want to find baseline points for a current(um) versus potential(V) graph.I am too interested in the method that you are using for your qPCR experiments.Thank you!

Answer 2

[Richard Brown]

Why don't you look for local maxima of curvature? This way you don't have to define any subjective tolerances. You may need to smooth your data first to make sure the finite difference derivative approximations work cleanly though.

To keep it simple, I'll assume your independent variable is evenly spaced and use a simple example (decaying exponential). We want to find the "knee"

dt = 0.01;
t = 0:dt:1;
y = exp(-10*t);

Compute first and second derivatives by finite differencing (centred)

yp = nan(size(y));
ypp = nan(size(y));
yp(2:end-1) = (y(3:end) - y(1:end-2))/(2*dt);
ypp(2:end-1) = (y(3:end) + y(1:end-2) - 2*y(2:end-1)) / (dt^2);

Compute the curvature

k = abs(ypp) ./ (1 + yp.^2).^1.5

Find the maximum and plot it on the curve. You could easily adapt this to find local maxima of k, etc.

[kmax, idx] = max(k);
plot(t, y, 'b', t(idx), y(idx), 'ro')

Also of interest to plot the curvature

figure()
plot(t, k)

edit: fixed wrong indices in plot command

[Nick]

The first plot statement uses the wrong indices for t and y. It should be:

plot(t, y, 'b', t(idx), y(idx), 'ro')

[Richard Brown]

@Nick, thanks. fixed now

Answer 3

[Sargondjani]

take the difference between each two points in the vector (use 'diff')

evaluate if these numbers 'grow faster'. Im not sure what your criterium is here. If you specify that, somebody might be able to help you further

[Sobhan]

Dear Sargondjani,

Thanks for your reply. I have used diff and as you said I can see exactly when differences grow faster. But I need a kind of formula to determine the exact point of acceleration for example it should tell me that x(15) or x(36) are turning points of my data-set.

I have hundreds of vectors like this.

Cheers

Sobhan

[Sargondjani]

my problem was that i dont know what your definition of acceleration is...

i mean, you say that the differences between points is too small to directly tell when there is change. this implies that some points are numerically EXACTLY the same and some are not. that means there is some sort of acceleration before your turning point as well

so you need to be more specific about what acceleration is...

Answer 4

[Sobhan]

Dear all, thanks for all your suggestions. Geoff`s answer helped me to solve the problem. As he suggested by defining a tolerance (specific to my data) I could find the points which I was interted in. Probably Richard`s solution also works but honestly it was a little complicated for me. I am really new to matlab ;) All the best, Sobhan

Answer 5

[Masoud Hosseiny]

Hi. you can calculate the differential of data set and plot it at the same time.

[Kriti Modi]

Even I have similar kind of data set and wondering I need to do complex stuff or if i can find it with simple maths. Explaining it here. I have a data set about part number and I want to divide the data set in three category low , medium and high or may be one more. I could see it visually in the graph but this data set is always changing. I want to automate the process mathematically to find these points every time data is changing. I think I need to see maximum rate of change? Is there any way to find it?