Create Semantic Segmentation Using Volume Segmenter

This example shows how to create a semantic segmentation of a volume using the Volume Segmenter app. The Volume Segmenter app offers many ways to explore a volume and segment objects in the volume. For example, you can view the volume slice-by-slice or as a 3-D representation. To segment an object, you can draw a region of interest (ROI) using ROI drawing tools or a paint brush tool. This example segments a stack of MRI images to label the brain and tumor regions. The example also labels the background.

Load Volumetric Data into the Workspace

Load a volume into the workspace. This example uses a stack of MRI brain images, stored in the MAT-file vol_001.mat. The MRI data is a modified subset of the BraTS data set [1].

load(fullfile(toolboxdir("images"),"imdata", ...
    "BrainMRILabeled","images","vol_001.mat"));
whos vol

  Name        Size                    Bytes  Class     Attributes

  vol       240x240x155            17856000  uint16              

Open the Volume Segmenter

Open the Volume Segmenter app. Click the Apps tab on the MATLAB® toolstrip. In the Image Processing and Computer Vision section, click Volume Segmenter.

Load the Volume into the Volume Segmenter

To load the volume in the Volume Segmenter app, click Open Volume in the app toolstrip. For this example, select Open from Workspace. In the Import Volume dialog box, select the volume you loaded into the workspace, vol, and click OK. Alternatively, you can specify a volume when you open the app by using the volumeSegmenter command:

volumeSegmenter(vol)

The Volume Segmenter app displays a 3-D representation of the volume in the 3-D Display pane and displays individual slices of the data set in the Slice pane.

By default, the Slice pane displays the first slice of your data. The app displays the number of the slice displayed at the top of the image, for example, 1/155. In this data set, the first few slices do not contain images of the brain.

The app also automatically creates a label for the segmentation in the Labels pane, using the default name Label1. You can define multiple labels in the Labels pane. However, to create a binary mask, you must use only one label.

To change the name of the label, double-click the label name. To change the color associated with the label, double-click the color square displayed in the Labels pane. You can optionally load an existing set of labels into the app using the Open Labels button.

Explore the Volume

To determine what you want to segment, explore the volume using the 3-D Display pane and the Slice pane.

In the 3-D Display pane, you can rotate the volume to examine the data from every angle, using the mouse. You can also customize the display of the volume in the 3-D Display tab in the app toolstrip. For example, if you have metadata that describes the relative size of the voxels, you can specify it in the Spatial Referencing part of the 3-D Display tab in the app toolstrip. To improve your view of the data, you can change the background color used in the 3-D display, modify the threshold and opacity of the display, and include orientation axes with the display, as shown in the figure below. With the brain MRI data, you can see the tumor in the temporal lobe that you want to segment.

You can also view each slice of the volume in the Slice pane. Use the slider at the bottom of the pane to move from slice to slice. You can see the tumor on slice 35 through slice 88. By default, the Slice pane displays the volume oriented along the X-Y axis, but you can change this using buttons in the Orientation section of the toolstrip on the Segmenter tab. The Slice pane is also where you use drawing tools to define the mask.

Use Drawing Tools to Label Regions in Volume

Once you have identified the object you want to segment, you can use the tools on the Draw tab in the app toolstrip to define the region. Select the drawing tool you want to use from the ROI tools: Freehand, Assisted Freehand, and Polygon, and a Paint Brush tool.

Start by labeling the brain. When one object is nested in another object, as the tumor appears over the brain on slices, label the larger region first. The first step is to create a label in the Labels pane. The app provides one label by default, named Label1. To change the name of the label to be more descriptive for your application, double-click on the label and type in the new name. To change the default color associated with the label, double-click on the colored square in the label identifier and select a color from the Color dialog box.

In the Slice pane, navigate to the slice where the object first appears and use a drawing tool to label the object. In the following figure, this example uses the Paint Brush tool to label the brain, but you can use any of the drawing tools.

Using Interpolation to Speed Object ROI Creation

You can move through the volume, slice-by-slice, and draw an ROI on each slice where the object appears. However, the Volume Segmenter app provides several automated interpolation tools that can help with segmenting an object across slices.

To use interpolation, you must first manually define the region on two slices. You have already defined the region on the first slice where the object appears, slice 35. Use the same process to define the region on the last slice where it appears, slice 88. The app places two bars on the slider, using the color associated with the label, to indicate the slices with ROIs.

With the ROI defined on two slices, click Auto Interpolate. The app automatically defines the ROI on all the intervening slides. The app uses blue bars to indicate all the slices that have ROIs, which now appear like a solid bar from slice 35 to slice 88.

Alternatively, after defining an ROI on two slices, you can click Manually Interpolate. With this option, the app opens the Manually Interpolate dialog box. You select the two regions from which you want to interpolate, Region One and Region Two. To select the first region, use the slider at the bottom of the dialog box to navigate to the first slice with an ROI, slice 35, and then click inside the ROI displayed. To select the second region, click Region Two, navigate to slide 88, and click inside the ROI displayed. After selecting both regions, click Run to interpolate the ROI on all intervening slices.

Refine the Interpolated Labels

After using interpolation, check the individual slices to see if the interpolation created satisfactory ROIs. Note that the ROI on slice 71 does not fill the entire object that you want to segment. You can manually adjust the ROI using the Paint Brush tool. Alternatively, you can use one of the tools in the Automate tab. For example, you can use Active Contours to grow the ROIs on the slices where it does not fill the full size of the tumor.

Perform Custom Processing

You can also add your own algorithm to operate on the ROIs. On the Automate tab, click Add Algorithm. Choose whether you want your processing function to operate on each 2-D slice (Slice-based) or on the entire 3-D volume (Volume-based).

For this example, under Slice-Based, select the New option and click Function Template to create a new function that operates on each 2-D slice. The app opens the template in the MATLAB editor. Replace the sample code in the template with code that you want to use. Your function must accept two arguments: each slice as a separate image and a mask. Your function must also return a mask image.

When you are done editing the template, save the file. The Volume Segmenter app automatically creates a button in the Automate tab toolstrip for your function. To test your function on one slice, click Run. By default, the app applies the function to only the current slice.

After testing your function on a single slice, you can run it on all of the slices or a subset of the slices. You can run it from the current slice to the end (the highest numbered slice) or from the current slice back to the beginning (slice 1). You can also specify a range of slices by specifying the starting slice and the ending slice.

When you choose one of the directional options, the app updates the slice numbers in the display. You can use this display to view the progress of processing.

Create Additional Labels

After labeling the brain on each slice, label the tumor wherever it appears on a slice, repeating the process described previously.

First, define a new label in the Labels pane. Click the Plus sign in the Labels pane to create a new label.

In the Slice pane, navigate to the slice where the object first appears and start labeling the object on each slice using a drawing tool. In the following figure, this example uses the Paint Brush tool to label the tumor. As previously, you can draw the object on each slice where it appears or use the interpolation tools to draw on multiple slices automatically. After interpolation, you can use drawing tools, such as the Eraser, to modify the automated segmentation on each slice.

Make the Background a Separately Labeled Region

When you define multiple labels, the app sets the data type of the label data as categorical. The default value for unlabeled categorical voxels is <undefined>. To label the background voxels so that they have a recognizable categorization, follow a similar process to that previously described:

1. Define a new label in the Labels pane, give the label a descriptive name, and select the color you want for the background.

2. Label the background on each slice. Navigate to a slice, select Fill Region on the Draw tab, and click anywhere in the background. Repeat this process on each slice.

When you add a background, it can obscure the other labels in the visualization of the volume in the 3-D Display pane. To view the other labeled regions in the 3-D Display pane, disable the visibility of the background label. Click Show Labels in the 3-D Display tab, click Customize, and deselect the visibility of the background label.

Save the Segmentation

When you complete labeling the brain and the tumor in the volume, save the segmentation. Click Save Labels on the Segmenter tab and choose from several options. You can save the labeled MRI data as a MAT file or as a variable in the workspace. For this example, choose a workspace variable and name the variable brain_labels.

After you save the segmentation, you can optionally turn on Autosave, which periodically saves the segmentation automatically.

View the Labeled Volume

To view the mask, use the volshow function. These commands demonstrate how to display the labels within the volume by adjusting the volume overlay properties:

viewer = viewer3d(BackgroundColor="white",BackgroundGradient="off",CameraZoom=1.5);
volDisp = volshow(vol,OverlayData=brain_labels,Parent=viewer, ...
    RenderingStyle="GradientOpacity",GradientOpacityValue=0.8, ...
    Alphamap=linspace(0,0.2,256),OverlayAlpha=0.8);

References

[1] Medical Segmentation Decathlon. "Brain Tumours." Tasks. Accessed May 10, 2018. http://medicaldecathlon.com/.

The BraTS data set is provided by Medical Segmentation Decathlon under the CC-BY-SA 4.0 license. All warranties and representations are disclaimed. See the license for details. MathWorks® has modified the subset of data used in this example. This example uses the MRI data of one scan from the original data set, saved to a MAT file.