Excluding noise from a curve
2 visualizzazioni (ultimi 30 giorni)
Hi, we have a dataset consisting of curves, similar to to the ones shown in the image below
we would like to set a criteria to help us get rid of the noise happening at the begging of the curve, i.e. the data before 200 on x-axis here.
we can't just exclude the data beofre 200 for all the curves since not all curves have this kind of noise and it is not neccessary that this noise apears always before 200, it can happen after 200 but not very much far.
We tried two things, one is to locate the first minimum in a curve then exclude the part before it. But again this will not work for all curves.
Second, we tried dealing with the first local maximum as an outlier and excluding the part according to it using rmoutliers(A) method but it was not detected as an outlier.
We need an advice here, how can we detect this part and remove it?
attached is the data for one of these curves.
Bjorn Gustavsson il 12 Apr 2023
Perhaps you will have to combine a number of criteria:
1, if you have a peak for x<300
2, and is above some threshold
3, and narrower than 150 in x-direction
But this type of removal of possible data malfunctioning can be a very difficult process to automate - what abou peaks that are "just failing" each of all sensible thresholds?