How to calculate the precision and recall ( determine FP and FN) for image detection based on image segmentation?

IF i used jaccard index and I got value of jaccrad =0.9 if I assume my threshold is 0.5 so if jaccard value >0.5 will be TP and if less will be FP

what about FN?

Answer 1

[Image Analyst]

Jaccard index is related to how much a "ground truth" binary image overlaps your "test" image. According to the help "similarity = jaccard(BW1,BW2) computes the intersection of binary images BW1 and BW2 divided by the union of BW1 and BW2, also known as the Jaccard index."

So using your definition, you're saying that if they overlap by more than 50%, it detected it correctly -- it's a True Positive. And you're saying that if they overlap by less than 50% it's a False Positive because even though they might overlap some, it's not enough to be considered correct.

A false negative would be if you said there was nothing there (gave a Negative) when there actually was something there.

Look at the different possible cases:

bw1 = true(2,2); % Something there.
bw2 = false(2,2); % Nothing there.
bwHalf = logical([1,1;0,0]);
similarityIndex = jaccard(bw1, bw1)

similarityIndex = 1

similarityIndex = jaccard(bw1, bw2)

similarityIndex = 0

similarityIndex = jaccard(bw2, bw1)

similarityIndex = 0

similarityIndex = jaccard(bw2, bw2)

similarityIndex = NaN

% Using half overlap
similarityIndex = jaccard(bw1, bwHalf)

similarityIndex = 0.5000

similarityIndex = jaccard(bw2, bwHalf)

similarityIndex = 0

Case 2 or 3, where the ground truth had something while the test/prediction said there was nothing, both give a Jaccard index of 0. That would be a False Negative -- it said it was negative but it was false because it should have said it was positive.

[yasmin ismail]

@Image Analyst you said 50% is not enough to consisederd to be correct, thus what threshold value should be considered to decide whether TP, FP?

[Image Analyst]

I didn't say that, you did. You said that if the overlap is over 50% it's a true positive: "I assume my threshold is 0.5 so if jaccard value >0.5 will be TP" Why you picked that amount is basically a judgment call that you decide on.

[yasmin ismail]

@Image Analyst

I got it .

about the recall

I know the recall =TP/(TP+FN)

Assume I have 4 images i got 3 of them TP, Iou>0.5 for the three images and fourth one is FN , what I know I have to calculate accumulative TP and accumulative FP

thus the precision for first image =1

second 2/2=1

third 3/3=1

fourth the precison=3/3=1

what about the Recall?some said the dominator will be total number of labeled image so if I have 4 images of crack as example the dominator always equal 4 but if I have one image FN from 4 images?

what i said is mentioned in Metrics on Object Detection. In this blog we will look into… | by gandham vignesh babu | Medium they applied accumalative in TP and FP , shall i applied accumalative FN too to calculate recall or direct the dominator of recall is the total number of images

Remember my condition is binary classification crack or not based on image segmentation

[Image Analyst]

Sorry for the delay but there is just so much you don't understand and it would take like an hour or two to explain it to you in person. It looks like you're trying to make an ROC curve. So start with understanding what that is:

https://en.wikipedia.org/wiki/Receiver_operating_characteristic

Now each individual point on an ROC curve (True Positive Rate vs False Positive Rate) corresponds to running your algorithm on the whole set of images. Each point corresponds to you changing something in your algorithm, such as the threshold, filter window width, or whatever it may be. So you set one value, run all your images through and then computer TP and FP and make that one point on the curve. Then you change the parameter to a new value and computer TP and FN for that new parameter. Now you have a second point. So then you repeat for as many parameter values as you want and you'll start building up the curve. Then you can examine the curve and decide what is the best parameter based on your situations. Obviously for some situations you'd want a really low FN rate (e.g. determination of cancer), while for other situations you might accept a lower FN or FP rate.

I still think that using Jaccard index itself to identify the best performing algorithm is sufficient. I don't see the rationale to threshold it and classify it into FP, FN, TP, and TN. You can just make a plot of Jaccard index vs your parameter value(s) and determine the best parameter(s) to use from that.

[yasmin ismail]

@Image Analyst I understod that , if i want to draw ROC i have to use accumulative for TP, FP, FN

But, if i want to just to calculate the precsion and recall for my model which is used detect crack based on basic image processing and segemntation?

I used the jaccard index to be able to determine TP, FP, FN then I can calculate the precision and recall for each image ( assume I have 100 images) without using acummulative for TP, FP, FN, then I can get the average of precision and recall , is it correct way?

[Image Analyst]

By your definitions, TP is the number of images where the jaccard index is more than 0.5. And FN is the number of images where there is actually a blob present but your algorithm says the jaccard index is less than 0.5, because it's saying there is no crack there but there actually is. I believe these are the definitions:

TP = says it's there (jaccard index > 0.5), and it is.

FP = says it's there (jaccard index > 0.5), but it's not there.

TN = says it's not there (jaccard index < 0.5) and it's not

FN = says it's not there (jaccard index < 0.5) but it is.

[yasmin ismail]

@Image Analyst I consider FN if (jaccard index=0) as you mentioned above last few days ( the cases)

and FP = (jaccard index< 0.5).

[yasmin ismail]

@Image Analyst and also how can i measure jaccard index for uncracked image as shown in attached image , I cannt created ground truth for it because its un cracked nothing to mark and at same time my model output is totally black image which an indication uncracked image, thus mean i cant use jaccard index to determine TN

[yasmin ismail]

@Image Analyst third question is , if I want to draw ROC curve can i change threshold for IoU (.5, 0.70,0.75) and find the precision and recall at each of them , or its wrong way and the threshold should be a parameter in the model and at each value of this parameter I have to calaculate IoU and then get precsison and recall and compare them with each value of parameter?

[Image Analyst]

I don't know that you can use the Jaccard index for a false negative. If the ground truth is not there, but your algorithm falsely says there is something there then you might just use this

FN = sum(binaryImage, 'all') > 0 && max(groundTruthBinaryImage) == 0;

Let's say you had 100 images. To build up the ROC curve, you'd have to compute the TP and FP rates (meaning the average over all 100 images) and then plot one point on the curve.

Then you'd change something in your algorithm/model such as the threshold but it could be anything, like a filter window width or whatever. Then you'd run all 100 images through the algorithm again and compute new rates. Now you have a second point on the ROC curve. Now do that as many times as you want to get as many points on the curve as you want.

See roc

[yasmin ismail]

@Image Analyst I understoodd the ROC

but the FN not yet, and I said I have not ground truth for TN

The cases you discussed above are :

bw1 = true(2,2); % Something there.

bw2 = false(2,2); % Nothing there.

bwHalf = logical([1,1;0,0]);

similarityIndex = jaccard(bw1, bw1)

similarityIndex = 1

similarityIndex = jaccard(bw1, bw2)

similarityIndex = 0

similarityIndex = jaccard(bw2, bw1)

similarityIndex = 0

similarityIndex = jaccard(bw2, bw2)

similarityIndex = NaN

% Using half overlap

similarityIndex = jaccard(bw1, bwHalf)

similarityIndex = 0.5000

similarityIndex = jaccard(bw2, bwHalf)

similarityIndex = 0

Case 2 or 3, where the ground truth had something while the test/prediction said there was nothing, both give a Jaccard index of 0. That would be a False Negative -- it said it was negative but it was false because it should have said it was positive.

That what you said about FN before , the similarity index =0, then you said :

FN = sum(binaryImage, 'all') > 0 && max(groundTruthBinaryImage) == 0;

I get confused and cannt understand it?

[yasmin ismail]

@Image Analyst can you clarify it to me please?