Image Segmentation


What Is Image Segmentation?

3 things you need to know

Image segmentation is a commonly used technique in digital image processing and analysis to partition an image into multiple parts or regions, often based on the characteristics of the pixels in the image. Image segmentation could involve separating foreground from background, or clustering regions of pixels based on similarities in color or shape. For example, a common application of image segmentation in medical imaging is to detect and label pixels in an image or voxels of a 3D volume that represent a tumor in a patient’s brain or other organs.

Why Image Segmentation Matters

Several algorithms and techniques for image segmentation have been developed over the years using domain-specific knowledge to effectively solve segmentation problems in that specific application area.  These applications include medical imaging, automated driving, video surveillance, and machine vision.

Medical Imaging

During medical diagnosis for cancer, pathologists stain body tissue with hematoxylin and eosin (H&E) to distinguish between tissue types. They then use an image segmentation technique called clustering to identify those tissue types in their images. Clustering is a method to separate groups of objects in a scene. The K-means clustering algorithm finds separations such that objects within each cluster are as close to each other as possible, and as far from other objects in other clusters as possible.

Using clustering to distinguish between tissue types (bottom) in an image of body tissue (top) stained with hematoxylin and eosin (H&E).

Autonomous Driving

When designing perception for autonomous vehicles, such as self-driving cars, semantic segmentation is popularly used to help the system identify and locate vehicles and other objects on the road.

Using semantic segmentation to associate each pixel of the image with a class label (such as car, road, sky, pedestrian, or bike).

How Image Segmentation Works

Image segmentation involves converting an image into a collection of regions of pixels that are represented by a mask or a labeled image. By dividing an image into segments, you can process only the important segments of the image instead of processing the entire image. 

A common technique is to look for abrupt discontinuities in pixel values, which typically indicate edges that define a region. 

Using thresholding to convert to a binary image to improve the legibility of the text in an image.

Another common approach is to detect similarities in the regions of an image. Some techniques that follow this approach are region growing, clustering, and thresholding.

Segmenting regions based on color values, shapes, or texture.

A variety of other approaches to perform image segmentation have been developed over the years using domain-specific knowledge to effectively solve segmentation problems in specific application areas. 

Image Segmentation with MATLAB

With MATLAB®, you can:

  • Use apps to interactively explore different segmentation techniques
  • Simplify image analysis workflows using built-in image segmentation algorithms
  • Perform deep learning for image segmentation

Using Apps to Interactively Threshold Images

Image Segmenter App

Using the interactive Image Segmenter app, you can iteratively try several methods to segment an image before achieving the desired result. For example, you can use the app to segment and further refine the results of an MRI image of a knee with different methods.

Using Image Segmenter App to interactively apply different segmentation techniques.

Color Thresholder App

This Color Thresholder app lets you apply thresholding to color images by manipulating the color of the images interactively, based on different color spaces. For example, you can use the Color Thresholder app to create a binary mask using point cloud controls for a color image.

Using Color Thresholder App to apply color segmentation to separate the birds in the foreground from the background.

Using a Variety of Image Segmentation Techniques

With functions in MATLAB and Image Processing Toolbox™, you can experiment and build expertise on the different image segmentation techniques, including thresholding, clustering, graph-based segmentation, and region growing.


Using Otsu’s method, imbinarize performs thresholding on a 2D or 3D grayscale image to create a binary image. To produce a binary image from an RGB color image, use rgb2gray to first convert it to a grayscale image.

Using thresholding to convert a color image to a binary image.


This technique lets you create a segmented labeled image using a specific clustering algorithm. Using K-means clustering–based segmentation, imsegkmeans segments an image into K number of clusters.

Clustering technique to separate out the patterned background on the floor.

Graph-Based Segmentation

Graph-based segmentation techniques like lazy-snapping enable you to segment an image into foreground and background regions. MATLAB lets you perform this segmentation on your image either programmatically (lazysnapping) or interactively using the Image Segmenter app.

Lazy-snapping to separate the foreground and background regions.

Using the Image Segmenter app to interactively apply graph-based segmentation.

Region Growing

Region growing is a simple region-based (also classified as a pixel-based) image segmentation method. A popularly used algorithm is activecontour, which examines neighboring pixels of initial seed points and determines iteratively whether the pixel neighbors should be added to the region. You can also perform this segmentation on images using the Image Segmenter app.

Using the Image Segmenter app to perform region growing.

Deep Learning for Image Segmentation

Using convolutional neural networks (CNNs), a deep learning technique called semantic segmentation lets you associate every pixel of an image with a class label. Applications for semantic segmentation include autonomous driving, industrial inspection, medical imaging, and satellite image analysis.

Schematic of semantic segmentation technique.

Using MATLAB, you can design and train semantic segmentation networks with a collection of images and their corresponding labeled images, and then use the trained network to label new images. To label the training images, you can use the Image Labeler, Video Labeler, or Ground Truth Labeler apps.

Using Ground Truth Labeler app to perform semantic segmentation.

Learn More About Image Segmentation

With the Image Segmentation app, you can preview how images will look after segmenting them with intensity-based approaches as well as techniques such as graph cut, circle finding, and region growing.
Learn how to perform color-based segmentation, refine image masks, and analyze regions using interactive apps.
Learn the high-level workflow for semantic segmentation using a deep learning network. Additionally, learn how the Image Labeler app can expedite your workflow for ground truth labeling at the pixel level.
Segment colors in an automated fashion using the L*a*b* color space and K-means clustering.
Detect a cell using edge detection and basic morphology.
Use watershed segmentation to separate touching objects in an image.
Get started with image processing in MATLAB by walking through an example. This video gives you an idea of the basic image processing and analysis features available in Image Processing Toolbox.