Evaluate semantic segmentation data set against ground truth
[___] = evaluateSemanticSegmentation(___,
computes semantic segmentation metrics using one or more
Name,Value pair arguments to control the evaluation.
Evaluate Semantic Segmentation Results
triangleImages data set has 100 test images with ground truth labels. Define the location of the data set.
dataSetDir = fullfile(toolboxdir('vision'),'visiondata','triangleImages');
Define the location of the test images.
testImagesDir = fullfile(dataSetDir,'testImages');
Define the location of the ground truth labels.
testLabelsDir = fullfile(dataSetDir,'testLabels');
Create an imageDatastore holding the test images.
imds = imageDatastore(testImagesDir);
Define the class names and their associated label IDs.
classNames = ["triangle","background"]; labelIDs = [255 0];
Create a pixelLabelDatastore holding the ground truth pixel labels for the test images.
pxdsTruth = pixelLabelDatastore(testLabelsDir,classNames,labelIDs);
Load a semantic segmentation network that has been trained on the training images of
net = load('triangleSegmentationNetwork'); net = net.net;
Run the network on the test images. Predicted labels are written to disk in a temporary directory and returned as a pixelLabelDatastore.
pxdsResults = semanticseg(imds,net,"WriteLocation",tempdir);
Running semantic segmentation network ------------------------------------- * Processed 100 images.
Evaluate the prediction results against the ground truth.
metrics = evaluateSemanticSegmentation(pxdsResults,pxdsTruth);
Evaluating semantic segmentation results ---------------------------------------- * Selected metrics: global accuracy, class accuracy, IoU, weighted IoU, BF score. * Processed 100 images. * Finalizing... Done. * Data set metrics: GlobalAccuracy MeanAccuracy MeanIoU WeightedIoU MeanBFScore ______________ ____________ _______ ___________ ___________ 0.90624 0.95085 0.61588 0.87529 0.40652
Display the properties of the
metrics = semanticSegmentationMetrics with properties: ConfusionMatrix: [2x2 table] NormalizedConfusionMatrix: [2x2 table] DataSetMetrics: [1x5 table] ClassMetrics: [2x3 table] ImageMetrics: [100x5 table]
Display the classification accuracy, the intersection over union, and the boundary F-1 score for each class. These values are stored in the
ans=2×3 table Accuracy IoU MeanBFScore ________ _______ ___________ triangle 1 0.33005 0.028664 background 0.9017 0.9017 0.78438
Display the normalized confusion matrix that is stored in the
ans=2×2 table triangle background ________ __________ triangle 4730 0 background 9601 88069
dsResults — Predicted pixel labels
PixelLabelDatastore object |
PixelLabelImageDatastore | cell array of datastore objects
Predicted pixel labels resulting from semantic segmentation, specified as
a datastore or a cell array of datastore objects.
dsResults can be any datastore that returns
categorical images, such as
dsResults) must return a
categorical array, a cell array, or a table. If the
read function returns a multicolumn cell array or
table, the second column must contain categorical arrays.
dsTruth — Ground truth pixel labels
PixelLabelDatastore object | cell array of
Ground truth pixel labels, specified as a datastore or a cell array of
dsResults can be any datastore that
returns categorical images, such as
dsTruth) must return a categorical
array, a cell array, or a table. If the
returns a multicolumn cell array or table, the second column must contain
imageSetConfusion — Confusion matrices for segmented images
table | cell array
Confusion matrix for the classes in the segmented images, specified as one of the following, where F is the number of images in the data set.
Table with F rows and one variable with the name
ConfusionMatrix. Each row in the table contains a cell array with the confusion matrix for the corresponding image.
F-by-one cell array. Each element of the cell array contains the confusion matrix for the corresponding image.
blockSetConfusion — Confusion matrices for segmented blocks
Confusion matrices for segmented blocks, specified as a table with
B rows and three columns, where B
is the total number of blocks in all images in the data set. The three
columns are the variables
You can obtain a table of the correct format by using the
segmentationConfusionMatrix function within a call to the
For an example, see Calculate Segmentation Metrics in Block-Based Workflow.
classNames — Class names
array of strings | cell array of character vectors
Class names, specified as an array of strings or a cell array of character vectors.
["sky" "grass" "building"
Specify optional pairs of arguments as
the argument name and
Value is the corresponding value.
Name-value arguments must appear after other arguments, but the order of the
pairs does not matter.
Before R2021a, use commas to separate each name and value, and enclose
Name in quotes.
computes only the mean BF score of each class, each image, and the entire data
Metrics — Segmentation metrics
"all" (default) | vector of strings
Segmentation metrics in
semanticSegmentationMetrics to compute, specified as the
comma-separated pair consisting of
'Metrics' and a
vector of strings. This argument specifies which variables in the
ImageMetrics tables to compute.
NormalizedConfusionMatrix are computed regardless
of the value of
|Value||Description||Aggregate Data Set Metric||Image Metric||Class Metric|
Evaluate all semantic segmentation metrics.
|All aggregate data set metrics||All image metrics||All class metrics|
Accuracy indicates the percentage of correctly identified pixels for each class. Use the accuracy metric if you want to know how well each class correctly identifies pixels.
The class accuracy is a simple metric analogous to global accuracy, but it can be misleading. For example, labeling all pixels "car" gives a perfect score for the "car" class (although not for the other classes). Use class accuracy in conjunction with IoU for a more complete evaluation of segmentation results.
The boundary F1 (BF) contour matching score indicates how well the predicted boundary of each class aligns with the true boundary. Use the BF score if you want a metric that tends to correlate better with human qualitative assessment than the IoU metric.
For more information, see
This metric is
not available when you specify a confusion matrix
Intersection over union (IoU), also known as the Jaccard similarity coefficient, is the most commonly used metric. Use the IoU metric if you want a statistical accuracy measurement that penalizes false positives.
For more information, see
|Average IoU of each class, weighted by the number of pixels in that class. Use this metric if images have disproportionally sized classes, to reduce the impact of errors in the small classes on the aggregate quality score.||none|
calculates the global accuracy and IoU metrics across the data set,
images, and classes.
Verbose — Flag to display evaluation progress
1 (default) |
Flag to display evaluation progress information in the command window,
specified as the comma-separated pair consisting of
'Verbose' and either
The displayed information includes a progress bar, elapsed time, estimated time remaining, and data set metrics.
metrics = evaluateSemanticSegmentation(pxdsResults,
pxdsTruth,'Verbose',0) calculates segmentation metrics
without displaying progress information.
ssm — Semantic segmentation metrics
Semantic segmentation metrics, returned as a
blockMetrics — Block-based semantic segmentation metrics
F-by-one cell array
Block-based semantic segmentation metrics, returned as an F-by-one cell array, where F is the number of images in the data set. Each element in the cell array contains information about all of the metrics calculated for all blocks in the corresponding image, formatted as a table.
Each table has K(f) rows, where K(f) is the number of blocks in the fth image in the data set. The table has up to five variables:
The table always includes the
BlockInfovariable. This table data in this variable are structs that provide spatial information about the block. The four fields of the struct are
DataEndWorld. For more information about these fields, see the IncludeBlockInfo name-value pair argument of the
The table includes the metrics in the DataSetMetrics property of the
ssmoutput argument. By default, the metrics are
WeightedIoU. However, if you create the
ssmand specify a subset of the metrics to calculate by using the
Metricsname-value pair argument, then the table includes only the specified metrics.
A value of
NaNin the dataset, class, or image metrics, indicates that one or more classes were missing during the computation of the metrics when using the
evaluateSemanticSegmentationfunction. In this case, the software was unable to accurately compute the metrics.
The missing classes can be found by looking at the
ClassMetricsproperty, which provides the metrics for each class. To more accurately evaluate your network, augment your ground truth with more data that includes the missing classes.
 Csurka, G., D. Larlus, and F. Perronnin. "What is a good evaluation measure for semantic segmentation?" Proceedings of the British Machine Vision Conference, 2013, pp. 32.1–32.11.
Automatic Parallel Support
Accelerate code by automatically running computation in parallel using Parallel Computing Toolbox™.
Introduced in R2017b
plotconfusion(Deep Learning Toolbox) |
- Calculate Segmentation Metrics in Block-Based Workflow
- Getting Started with Semantic Segmentation Using Deep Learning
- Deep Learning in MATLAB (Deep Learning Toolbox)
- Datastores for Deep Learning (Deep Learning Toolbox)