How do I find the groundtruth of an image?

I'm trying to evaluate the accuracy of my detection results. May I know how do I label the groundtruth and do I have to manually label the object? If so, how do I do it?

Answer 1

[Dima Lisin]

Try the Training Image Labeler app in the Computer Vision System Toolbox.

[Leonard Yeo]

How do I do that?

[Dima Lisin]

First you need to have the Computer Vision System Toolbox installed on your machine with a current license. Then you start MATLAB and type

>> trainingImageLabeler

Then you load your images and start drawing bounding boxes around objects of interest with your mouse.

[Leonard Yeo]

I have done that for my training images. Do I repeat the same thing for my test images set? Because the goal for me is to evaluate the accuracy of my result against the groundtruth. So I need to know how to label the groudtruth in order for me to compare 2 bounding box (One of them is the groundtruth and the other is the result of my work) with each other.

[Image Analyst]

No you don't, though you could if you know what ground truth is for those images. The usual process is you train it, or develop your algorithm, with some limited number of ground truth images, then determine the accuracy, then you can then run through many, many more images, for which you don't know ground truth for, and assume the accuracy is the same because you developed/trained your algorithm with a representative sample. You don't have ground truth for all your images, do you? If so, there's no need to develop or train anything - you already know the answer. The only reason to develop an algorithm is if you don't know what the result would be.

[Dima Lisin]

Hi Leonard,

Typically, you label your objects of interest in a large set of images. Then you use some of these images (e. g. 80%) for training. Then you evaluate your detector on the remaining images, which you did not use for training.

To do the evaluation, you run your detector on a labeled image, and then see how much your detected bounding boxes overlap with the ground truth (i. e. the hand-labeled bounding boxes). You can use the bboxOverlapRatio function to see how much one box overlap with another. Then you would need to choose some threshold to decide how much overlap is enough for the detection to count as correct.

[Leonard Yeo]

Does "hand-labeled bounding boxes" refers to manually draw a bounding box around the object using any kind of program (i.e. paint or photoshop)?

If so, does it mean that I will then use bboxOverlapRation to compute the difference between the groundtruth and detected box? But how do I know the [x y width height] of the groundtruth bbox if I draw it manually?

[Dima Lisin]

"Hand-labeled bounding box" means you draw it by hand, and somehow extract its coordinates. This is exactly what trainingImageLabeler lets you do, and you said you were using it already.

When you labeled your training images, you have created ground truth. You can simply use some of those images for testing.

[Walter Roberson]

Semantically, ground truth is what is "really" there, potentially more accurate than your labeling of it from a visual image. You might need to use multiple modalities to find ground truth, such as combining x-ray with MRI and contrast-enhanced florescence. The whole point can be to see how well you can match reality by using a single or inexpensive or faster method and good algorithms.

Answer 2

[Walter Roberson]

Someone has to manually label the ground truth. That might occur by using some kind of program to do the "obvious" parts of it, followed by a human touching it up.

Ground truth labeling is, by definition, only applied to labeling that is absolutely completely correct in every location, known to be correct because of some external information. For example, a picture of a boundary might have been labeled after electron microscope examination of the surface in order to determine atom by atom where one substance stopped and another started. For a picture of a forest, you might have had a team of scientists go tree by tree extracting DNA samples and then having specialists in each genus determining exactly which species (or varietal) occurred.

If there is any chance of error in a groundtruth labeling then the labeling is not groundtruth -- but it might be Gold Standard (determined by an experienced expert) instead. Groundtruth is "we know for certain that this is right".

[Leonard Yeo]

" Someone has to manually label the ground truth." do you mean that I have to manually draw a bounding box around the object using any kind of program (i.e. paint or photoshop)?

[Walter Roberson]

No, but that is one option. Bounding boxes get clumsy for anything that is curved or irregular shaped.