Interactively Segment Image Using Segment Anything Model

This example shows how to perform interactive segmentation of an object in a selected region of interest (ROI) of an image using the Segment Anything Model (SAM).

The SAM is a zero-shot image segmentation model that uses deep learning neural networks to accurately segment objects within images without requiring training [1]. The SAM enables you to actively guide and refine segmentation by providing feedback through visual prompts, such as points, boxes, and mask logits. In this example, the SAM uses the ROIs you interactively select on the image as the visual prompts. To get started with the SAM, see Get Started with Segment Anything Model for Image Segmentation.

This example requires the Image Processing Toolbox™ Model for Segment Anything Model. You can install the Image Processing Toolbox Model for Segment Anything Model from Add-On Explorer. For more information about installing add-ons, see Get and Manage Add-Ons. The Image Processing Toolbox Model for Segment Anything Model requires Deep Learning Toolbox™ and Computer Vision Toolbox™.

Load Sample Image

Load the sample image into the workspace.

I = imread("peppers.png");

For best SAM performance, use an image with a data range of [0, 255], such as one with a uint8 data type. If your input image has a larger data range, rescale the range of pixel values using the rescale function.

Create Model

Create a SAM using the segmentAnythingModel object.

samObj = segmentAnythingModel;

Extract Embeddings

To decrease computation time, use GPU processing, if it is available.

if(canUseGPU)
    I = gpuArray(I);
end

Extract the feature embeddings of the input image I from the encoder of the model using the extractEmbeddings object function.

embeddings = extractEmbeddings(samObj,I);

Configure Interactive Interface

Prepare the image display for SAM interactive segmentation. Create a new figure window and axes, and display the image.

f = figure;
hAX = axes(f);
hIm = imshow(I,Parent=hAX);

Specify an ROI that contains an object to segment as a Rectangle object on the image by using the drawrectangle function. Draw a rectangular ROI around the object you want to segment in the image display.

roi = drawrectangle(hAX);

Interactively Segment Object in Image

To segment the object, run the SAM decoder on the image feature embeddings using the segmentObjectsFromEmbeddings object function. Specify the bounding box visual prompt BoundingBox as the position of the rectangle ROI that you interactively drew on the image display.

mask = segmentObjectsFromEmbeddings(samObj,embeddings,size(I),BoundingBox=roi.Position);

Visualize the segmentation mask overlaid on the image in the image display by using the displayMask helper function.

displayMask(hIm,I,mask)

Interactively Segment Object in Movable ROI

To automatically segment an object every time you move or resize an ROI around it on the image display, configure a listener for the moving ROI event by using the addlistener function. Specify the listener callback using the segmentObjectInROI helper function.

addlistener(roi,"ROIMoved",@(src,evt)segmentObjectInROI(evt,samObj,embeddings,I,hIm))

Move or resize the ROI around different objects in the image to segment each of them. This GIF shows how to segment individual objects, one at a time, by moving and resizing the ROI around each one.

Supporting Functions

segmentObjectInROI

Performs object segmentation at the current position of the specified ROI, and displays the segmentation mask on the image.

function segmentObjectInROI(evt,samObj,embeddings,img,hIm)
    % Segment object in the current ROI.
    boxPrompt = evt.CurrentPosition;
    mask = segmentObjectsFromEmbeddings(samObj,embeddings,size(img),BoundingBox=boxPrompt);
    % Display the segmented mask overlayed on the original image.
    displayMask(hIm,img,mask); 
end

displayMask

Displays the object mask overlaid on the image.

function displayMask(hIm,img,mask)
    overlayedImg = insertObjectMask(img,mask);
    hIm.CData = overlayedImg; 
end

References

[1] Kirillov, Alexander, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, et al. "Segment Anything," April 5, 2023. https://doi.org/10.48550/arXiv.2304.02643.