How to determine the threshold for SSIM of images?

I created detected crack model based on image segmenation and ssim value for my images 0.95 and above , so how to decide the threshold?

I attached the original image 7027-157 and the groundTruth(label7027-157) and the output image (new7027-157) from the created model (which is created to detect crack) to compaire with groundTruth. so if I have 100 images how can i determine the threshold for ssim?

Answer 1

[Image Analyst]

You can set the threshold for whatever you want. I have no idea how you'd even use it. I'm not sure what you did. You cannot sensibly use ssim to compare any of those three images to each other - it just doesn't make sense. Just think about it. They're far too different. So setting a threshold for it would also not make sense. At best you might judge how good a job your segmenter did by comparing your ground truth to the "new" image, but even that is not good. That's because your "new Image" doesn't look like your ground truth image. It looks like it's been skeletonized so you'd have to use something like dice to compare the skeleton of your ground truth image to the "new" skeletonized image. Or else you can use dice to compare a binarized version of the ground truth image to the "new" image before you binarized it.

Again, ssim would not be used here. It's meant for comparing two similar images, such as a gray scale image and one that underwent a round trip compression and decompression for example.

[yasmin ismail]

@Image Analyst I created the groundtruth by using segmenter app, I used ROI , assisstand freehand then export to workspace. when I used dice I got very small result as shown

A = logical(imread('new7027-157.png'));;

BW_groundTruth =logical(imread('Label7027_157.png'));

similarity = dice(A, BW_groundTruth)

similarity =

0.4402

while when I used ssim

A = imread('Label7027_157.png');

BW_groundTruth =imread('new7027-157.png');

[ssimval,ssimmap] = ssim(A,BW_groundTruth);

imshow(ssimmap,[])

title("Local SSIM Map with Global SSIM Value: "+num2str(ssimval))

0.93

So how to improve it using dice?

[Image Analyst]

Your dice score is so low because the new image is only a skeleton, while the ground truth gray scale image, when you call logical(), converts all the non-zero pixels to true so it's a lot bigger. If youi're going to compare thick cracks then both should be thick cracks. If you're going to compare skeletons, then both should be skeletons. You can't mix them.

[yasmin ismail]

@Image Analyst if both skeleton, after labeleing by image segmenter app I applied the following to skeletonize it

im = imread('Label7027_157.png');

im_gray2 = rgb2gray(im);

im_bin2 = imbinarize(im_gray2);

skelImage = bwskel(im_bin2);

Then I applied dice but i got also small similarity:

BW_groundTruth = skelImage;

A =imread('new7027-157.png');

AA=rgb2gray(A);

AAA = imbinarize(AA);

similarity = dice(AAA, BW_groundTruth)

similarity =

0.2021

It is still small too, although both are skeleton, is there something wrong I did?

@Image Analyst

[Image Analyst]

Youi'll have to zoom in and look at each pixel of the skeletons and see if they are both white at the same location. Even if it's off by just one pixel, it will affect the result. It looks like maybe 80% of your pixels are off.

[yasmin ismail]

@Image Analyst I understand that, but the problem is I used freehand assesstant for ROI in image segmenter app to label crack as shown in attached images(screenshot from Segmenter App with and without binary ). Is it incorrect way?, because this way not let each pixel from two images (groundTruth(label7027-157) and the (new7027-157)) at the same location, thus how to let both at same location ?

Note: I skelotonize labeled image after export it work space as shown above

[Image Analyst]

You do realize that if you train a model with training images and ground truth images, and then use that trained model and run through your training images that you will not get exactly the corresponding ground truth image out, right? The more images you have and the longer you train, the closer to the ground truth you will get, but you don't want to train overly long so that you get your ground truth images out exactly, because in that case, it's merely memorizing your input images and your predicted image for an image not in your training set will quite possibly be worse than if you didn't train as long. This is the classic problem of overfitting.

[yasmin ismail]

@Image Analyst I didnt understand ,

1- did I labeled image correctly? and

2-if I want to compaire between groundTruth and the predicted image I have to use semantic segementation to train it, thats what you mean above?

3-can I comapire between each image and its groundTruth seperatly using dice or jaccard to calculate the precision and recall?

4- if the most of dice scores for each image around 0.2 ,then can i decide the threshold to be 0.2 so above it will be true positive , lower than 0.2 false postive so at the end I can calculate the precision and recall based on the steps I applied above ( labeled image image by segmenter app, then skelotinze labeled image, then use dice)?

[Image Analyst]

1. Yes, by definition, you labeled them correctly because there is no other source of ground truth other than your opinion so however you labeled them, in your mind, it was labeled correctly. How could they not be?
2. Yes, semantic segmentation would be the way to get your deep learning neural network model predict what you labeled, though the prediction will not match exactly of course.
3. If you labeled the region(s) and trained the model, and then took one of your training images and ran it through the model, then it should create a binary image mask that is where it thinks the region is in that image. To see how well it matches up with your ground truth image you need to get a binary image of your ground truth image and that would be a binary image of the region. You can use ismember to extract the pixels with the label number, mask = ismember(labeledImage, labelNumber), or you can use simple thresholding. I would not skeletonize anything at this point. After you're satisfied that the prediction is good, then as part of subsequent analysis you might want to skeletonize, for example to get the crack length in addition to the area.
4. You can compute the dice scores for your training images. What the threshold for acceptable or not acceptable is, is up to you. You may decide that if it's more than 90% overlap between ground truth and predicted that it's good enough to start using. Or you may decide that it's not good enough and you'll need to train more, either longer or (preferably) with more training images.

[yasmin ismail]

@Image Analyst shall I have to train the model , its required to creat deep learning model and I dont know how to creat it, so if I have 100 images can I compaire each image (groundTruth and predicted) and get the similarity scores for 100 images by the following code for each image:

similarity = jaccard(A(:,:,1), BW_groundTruth)

then written the result for each on excel sheet , then calculate the precision and recall by chosing the threshold to decide TP and FP?

[Image Analyst]

See attached general purpose app that I made to train the network and predict masks using the trained network.

[yasmin ismail]

@Image Analyst I appreciate your help thanks alot , first I clicked on PredictMask.fig then the Interface is opened, then I click on Step 1 Select RGB Image folder and i got the following error :

Error in predictMask (line 42)

gui_mainfcn(gui_State, varargin{:});

Error in

matlab.graphics.internal.figfile.FigFile/read>@(hObject,eventdata)predictMask('btnSelectRGBFolder_Callback',hObject,eventdata,guidata(hObject))

Error while evaluating UIControl Callback.

How it works?

[Image Analyst]

You need to run the training app first and save a model .mat file. Then you can use the predict app to read that model and apply it to images.

It's possible some of my utilities might be called so you should save the utilities folder in the attachment and set a path to it.