Create Labels and Label Multi-Sensor Data

R2026b

After loading signals into the Multi-Sensor Labeler app, as described in the Load and View Multi-Sensor Data procedure, create label definitions and label the signal frames.

Create ROI Label Definitions

Label definitions contain the information about the labels that you mark on the signals. You can create label definitions interactively within the app or programmatically by using a labelDefinitionCreatorMultiSensor object.

An ROI label is a label that corresponds to a region of interest (ROI) in a signal frame. You can define these ROI label types in the Multi-Sensor Labeler app:

  • Rectangle/Cuboid — Draw axis-aligned bounding box labels around objects, such as vehicles. In image signals, you draw labels of this type as 2-D rectangular bounding boxes. In point cloud signals, you draw labels of this type as 3-D cuboid bounding boxes.

  • Rotated Rectangle — Draw 2-D bounding box labels around objects in an image that are rotated.

  • Point — Draw a point label in an image signal to identify an object.

  • Line — Draw linear ROIs to label lines, such as lane boundaries. Supported in both image and point cloud signals.

  • Projected Cuboid — Draw 3-D bounding box labels around objects in an image, such as vehicles.

  • Polygon — Draw polygon labels around objects. You can label distinct instances of the same class.

  • Pixel — Draw pixels to label various classes, such as road or sky, for semantic segmentation of image data.

  • Semantic Point — Label individual points in a point cloud for per-point semantic classification. This label type was previously named Voxel ROI. For more details, see Semantic Point Labeling.

Create an ROI label definition:

  1. In the Labeler tab, from the toolstrip, click Add Label and select the desired label type.

  2. In the Define New Label dialog box, specify a Label Name.

  3. From the Group list, select New Group and name the group. Adding labels to groups is optional.

  4. Click OK. The group name appears on the ROI Label Definitions pane with the label name under it.

ROI Labels tab with a Vehicles group that contains a car label

A Rectangle/Cuboid label is drawn differently on each signal type. On image signals, the label is drawn as a 2-D rectangular bounding box of type Rectangle. On point cloud signals, the label is drawn as a 3-D cuboid bounding box of type Cuboid.

Create ROI Sublabel

A sublabel is a type of ROI label that corresponds to a parent ROI label. Each sublabel must belong to, or be a child of, a label definition that is in the ROI Label Definitions pane. For example, in a driving scene, a vehicle label can have sublabels for headlights, license plates, or wheels.

Create an ROI sublabel definition:

  1. Select the parent label of the sublabel. On the ROI Label Definitions pane in the left pane, click the parent label to select it.

  2. In the Labeler tab, from the toolstrip, click Sublabel, and select the sublabel type (for example, Rectangle).

  3. In the Define New Sublabel dialog box, specify a Sublabel Name. Click OK.

The sublabel appears in the ROI Label Definitions pane under the parent label. The sublabel and parent label have the same color.

ROI Labels tab with a brakeLight sublabel under the car label

Note

Sublabels are supported for Rectangle, Rotated Rectangle, Line, Polygon and Projected Cuboid label types on image signals only. Cuboid, Pixel, Point and Semantic Point labels do not support sublabels.

Create ROI Attribute

An ROI attribute specifies additional information about an ROI label or sublabel. For example, in a driving scene, attributes can include the type or color of a vehicle. You can define ROI attributes of these types:

  • List — Specify a drop-down list attribute of predefined strings, such as make or model of a vehicle.

  • Numeric — Specify a numeric scalar attribute, such as the number of doors on a labeled vehicle.

  • String — Specify a string scalar attribute, such as the color of a vehicle.

  • Logical — Specify a logical true or false attribute, such as whether a vehicle is in motion.

Create an attribute:

  1. On the ROI Label Definitions pane, select the label or sublabel to which you want to add the attribute.

  2. In the Labeler tab, from the toolstrip, click Attribute, and select the attribute type from the list.

  3. In the Define New Attribute dialog box, specify an Attribute Name. Optionally set a default value and provide a description. Click OK.

    ROI Labels tab with an isOn attribute under the brakeLight label

Create Scene Label

A scene label defines additional information across all signals in a scene. Use scene labels to describe conditions, such as lighting and weather, or events, such as lane changes.

Create a scene label definition:

  1. In the Labeler tab, from the toolstrip, click Add Label, and select Scene from the list.

  2. In the Define new scene label dialog box, specify Label Name and Type. Scene labels can either be logical, numeric or string values.

  3. Choose a Color for the label by clicking the color preview and selecting a color.

    Define new scene label window configured to display the daytime label.

  4. Optionally, from the Group list, select New Group and name the group.

  5. Click OK. The Scene Label Definitions pane shows the scene label definition.

Verify Label Definitions

Verify that your label definitions are set up correctly:

  1. The ROI Label Definitions pane contains the expected groups and labels with the correct types.

  2. Sublabels appear under their parent labels.

  3. Attributes appear under their associated labels or sublabels.

  4. The Scene Label Definitions pane contains the expected scene labels.

  5. Semantic Point ROI labels appear with the correct label names for per-point classification.

To edit or delete a label or sublabel definition, select the desired label or sublabel definition. Then, from the app toolstrip, click Edit or Delete.

To edit or delete an attribute, right-click that attribute and select the appropriate edit or delete option.

To save these label definitions to a MAT-file for use in future labeling sessions, on the app toolstrip, first select Export. Then, in the Label Definitions section, select To File.

To reorder label definitions or move them to different groups, you can drag and drop them in the label definition panes.

Label Video Using Automation

Note

Labeling Images and Videos in Multi-Sensor-Labeler app requires Computer Vision Toolbox™.

Use the built-in automation algorithms to label objects in video signals. For example, use the Temporal Interpolator to label a car across multiple frames:

  1. Select the time range to label. In the text boxes below the video, specify the start and end times for the interval. The app sets the range slider and text boxes and displays signal frames only from the specified interval. The red flags indicate the start and end of the interval. You can also click the blue interval icon below the slider to expand the time range to fill the entire playback section.

    Range slider configured to automate the 8 to 10 second interval. The left red flag is at second 8, and the right red flag is at second 10.

  2. In the ROI Label Definitions pane, click the label definition you want to automate.

  3. From Automate Labeling section of the app toolstrip, select Select Algorithm > Temporal Interpolator. This algorithm estimates rectangle ROIs between image frames by interpolating the ROI locations across the time range.

  4. Click Select Signals. In the Select Signals window, select only the video signal and click OK. This algorithm supports labeling of only one signal at a time.

  5. Click Automate. An automation session for the video opens.

  6. At the start of the time range, click and drag to draw a label around the target object. For this algorithm, you can draw only one label per frame.

  7. Drag the slider to the last frame and draw the same label around the same object. Optionally, label intermediate frames to improve results.

  8. From the Automate tab on the toolstrip, click Run. The automation algorithm applies the label to the intermediate frames.

  9. When satisfied with the results, click Accept to close the session and apply the labels.

    Video with the rectangular car label applied to a car in the center of the frame

By default, labels appear only when you move your pointer over them. To always display labels, from the Visualization tab on the app toolstrip, set Show ROI Labels to Always.

Label Point Cloud Using Automation

Use automation algorithms designed for point cloud labeling, such as the Point Cloud Temporal Interpolator:

  1. In the labeling window, click the point cloud signal to select it.

  2. In the ROI Label Definitions pane, click the label definition (for example, a Rectangle/Cuboid label).

  3. From Automate Labeling section of the app toolstrip , select Select Algorithm > Point Cloud Temporal Interpolator. This algorithm estimates cuboid ROIs between point cloud frames by interpolating the ROI locations across the time range.

  4. Click Select Signals, select only the point cloud signal, and click OK.

  5. Click Automate. An automation session for the point cloud opens.

  6. At the start of the time range, draw a cuboid label around the target object. . Click the point cloud signal frame to create the label and zoom in using the scroll wheel.

    Point cloud frame zoomed in on a car

    The label snaps to the highlighted portion of the point cloud. Adjust the cuboid until it fully encloses the object. To resize the cuboid, click and drag one of the cuboid faces. To move the cuboid, hold Shift and click and drag one of the cuboid faces.

    Cuboid car label applied to the vehicle. The cuboid is wider than the vehicle

  7. Use projected view to adjust the cuboid label in top-view, side-view and front-view simultaneously. Under the Point Cloud tab in the app toolstrip, select the Projected View option from the Camera View section.

  8. Copy the cuboid (Ctrl+C) and paste it (Ctrl+V) into the last frame. Optionally adjust the position.

  9. From the Automate tab, click Run. The algorithm applies the label to intermediate frames.

  10. Click Accept to close the session and apply the labels.

Refine and Adjust Point Cloud Labels

After drawing or automating cuboid labels, use the refinement tools on the Point Cloud tab to improve label accuracy. These tools help you precisely fit labels to sparse 3-D data points.

Projected View

Select Projected View to view the front-view, top-view, and side-view of the selected label simultaneously. Use these views to manually adjust the position and size of your cuboid labels from multiple perspectives.

Projected view for cuboid label adjustment

Auto Align Labels

Select Auto Align to fit a cuboid accurately to the label data and align the label in the direction of the object. This feature centers and orients the cuboid on the labeled point cluster for improved accuracy.

Label without Auto Align optionLabel with Auto Align option

Without auto align, the label box is neither centered on nor aligned with the object it labels.

With auto align, the label box is centered on and aligned with the object it labels.

Snap and Shrink Labels

On the Point Cloud tab, use these options to snap and shrink your labels to fit the points.

  • Shrink To Fit — Shrink the size of cuboid labels to best fit the labeled points.

  • Snap To Cluster — Snap labels to best fit the selected point clusters. You can also specify which clustering algorithm to use to fine-tune the point clusters, by selecting Cluster Settings.

    Cluster Settings Dialog Box

    • Select Range-based clustering to cluster point cloud data using the segmentLidarData function. This option is not applicable for unorganized point cloud data.

    • Select Distance-based clustering to cluster point cloud data using the pcsegdist function.

    You can specify the algorithm parameters in the Cluster Settings dialog box. To visualize the output point cloud clusters, select the View Clusters parameter.

  • Snap to Point — Snap the line label vertex to the nearest point in the point cloud.

Tip

For more visualization options, see Visualize Point Cloud Data.

Semantic Point Labeling

When you define a Semantic Point ROI label, the app opens the Semantic Point tab. You can draw and fine-tune the semantic point regions on a point cloud to label them.

Toolstrip options for semantic point tab.

You can manually label a semantic point region using these options:

  • Brush — To label regions using a paintbrush method, click Brush, and select the desired size of marker using the Brush Size slider. Then, drag the brush over the desired points in the point cloud data. To erase labels, click Brush Erase, adjust the Brush Size slider, and drag the eraser over the desired points.

  • Lasso — To label regions by outlining them using a free form lasso, click Lasso, and draw a region of interest (ROI) on the point cloud. To erase labels, click Lasso Erase. Then, draw an ROI on the point cloud.

Click Select Cluster to label a cluster of points. You can tune the clustering algorithm by selecting Cluster Settings. You can specify distance-based or range-based clustering.

You can control which data to show or hide in semantic point regions on a labeled point cloud by specifying the Display Data setting with one of these options:

  • All — Display all data.

  • Unlabeled — Display only unlabeled data. The app hides labeled data.

  • Labeled — Display only labeled data. The app hides unlabeled data.

  • Selected Label — Display only selected labeled data.

To update the visualization of data after any modification, click Update Display. For example, if you want to focus on unlabeled data for further labeling, first select Unlabeled. Then, label data and click Refresh to hide the newly labeled data and focus on the remaining unlabeled data.

Use the Semantic Point Opacity slider to adjust the transparency of the semantic point labels for better visualization.

Tip

To undo or redo a labeling action in the Semantic Point tab, press Ctrl+Z or Ctrl+Y, respectively.

Burst Mode Labeling

Burst Mode is a semi-automated labeling feature for point cloud signals that enables you to label static or semi-static objects across multiple timestamps at once. Burst mode operates on one selected point cloud signal, merges point clouds across a selected time range into a single view, and applies annotations across all timestamps in that range. It supports Cuboid and Semantic Point ROI labels only.

To use burst mode, follow these steps:

  • Select a point cloud signal and a Cuboid or Semantic Point ROI label definition.

  • Click Burst Mode in the range slider pane.

    Burst mode tooltip in the slider range pane.

  • Draw labels on the merged point cloud.

  • Click Accept and Exit to apply labels across all timestamps, or Exit to discard changes.

For detailed information on burst mode workflow, entry conditions, and UI behavior, see the Burst Mode section in Automate Labeling for Multi-Sensor Data.

Label with Sublabels and Attributes Manually

Manually label a frame of a video signal with sublabels and attributes. Point cloud signals do not support sublabels and attributes for cuboid or semantic point labels.

  1. Verify the time range is set to the desired interval.

  2. In the ROI Label Definitions tab, click the sublabel definition to select it.

  3. If needed, hide the point cloud signal. On the Labeler tab of the app toolstrip, in the Layout section, under Show/Hide Signals, clear the check mark for the point cloud signal. Hiding a signal only hides the display. The app maintains the labels for hidden signals, and you can still export them.

  4. In the image frame, select the parent ROI label. The label turns yellow. You must select the parent label before you can add a sublabel to it.

  5. Draw sublabels on the target regions within the parent label.

    A vehicle labeled "car" with rectangular "brakeLight" subabels applied to its brake lights.

  6. On the right, the View Labels, Sublabels and Attributes pane shows all the ROI and scene labels, sublabels, and their corresponding attributes present in the current frame. Set the attribute values as needed.

    Label, Sublabel and attribute pane showing car and brakeLight labels with isOn attribute set to false.

Label Scene Manually

Apply a scene label to the signal frames:

  1. Expand the time range to the desired duration. Drag the red flags to set the start and end of the range slider.

  2. From the Scene Label Definitions tab, select the scene label definition.

  3. In the View Labels, Sublabels and Attributes, select the appropriate scene label value. Click Apply to Time Range. A check mark appears for the scene label indicating that the label now applies to all frames in the time range.

Scene label pane with Apply to time range button.

View Label Summary

With labels applied to at least one frame of a signal, you can view a visual summary of the ground truth labels. On the Labeler tab of the app toolstrip, click View Label Summary. For logical scene labels, false indicates absence of labels whereas for numeric and string scene labels, an empty value indicates the absence of that labels.

Save App Session

Save your labeling session to preserve the data source, label definitions, and labeled ground truth data. The saved session also includes your session preferences, such as the layout of the app.

On the app toolstrip, you can manage sessions using these options:

  • Select Save Project, and then either Save or Save As, to save the current labeling session.

  • Click New Project to start a new session. In the dialog box that opens, specify a session folder.

  • Click Open Project, and select one of the listed recent sessions, or navigate to the folder of a previous session.

You can now either close the app session or continue to the Automate Labeling for Multi-Sensor Data step or the Export Multi-Sensor Ground Truth and Create Training Data step.

Programmatic Label Creation and Import

You can create label definitions and label data programmatically, then import them into the Multi-Sensor Labeler app. This workflow is useful for reusing label definitions across sessions or importing labels generated by external algorithms.

To programmatically create and import ground truth data:

  1. Create label definitions using the labelDefinitionCreatorMultiSensor object. Add ROI labels, sublabels, attributes, and scene labels as needed.

  2. Create data sources for each signal using the appropriate source class (for example, vision.labeler.loading.VideoSource or vision.labeler.loading.PointCloudSequenceSource).

  3. Create label data as timetables for each signal with timestamps as row times and label names as variable names. Store the label data in a vision.labeler.labeldata.ROILabelData object and scene label data in a vision.labeler.labeldata.SceneLabelData object.

  4. Construct a groundTruthMultiSensor object from the data sources, label definitions, ROI label data, and scene label data.

  5. Import the ground truth into the app. On the Labeler tab, click Import > Labels, and select From Workspace or From File.

To import only label definitions without label data, on the Labeler tab, click Import > Label Definitions.

For the complete programmatic workflow with code examples, see groundTruthMultiSensor.

See Also

| |

Topics