Apply Labels to Unreal Scene Elements for Semantic Segmentation and Object Detection
The Simulation 3D Camera block provides an option to output semantic segmentation data from a scene. If you add new scene elements, or assets (such as traffic signs or roads), to a custom scene, then in the Unreal® Editor, you must apply the correct ID to that element. This ID is known as a stencil ID. Without the correct stencil ID applied, the Simulation 3D Camera block does not recognize the scene element and does not display semantic segmentation data for it.
For example, this To Video Display window shows a stop sign that was added to a custom scene. The Semantic Segmentation Display window does not display the stop sign, because the stop sign is missing a stencil ID.
You can also use these stencil ID labels to assign IDs to objects detected by probabilistic sensors, such as those modeled by the Simulation 3D Probabilistic Radar and Simulation 3D Vision Detection Generator blocks.
To apply a stencil ID label to a scene element, follow these steps:
Open the Unreal Editor from a Simulink® model that is configured to simulate in the 3D environment. For more details, see Customize Scenes Using Simulink and Unreal Editor.
In the editor window, select the scene element with the missing stencil ID.
On the Details pane on the right, in the Rendering section, select Render CustomDepth Pass.
If you do not see this option, click the Show Advanced expander to show all rendering options.
In the CustomDepth Stencil Value box, enter the stencil ID that corresponds to the asset. If you are adding an asset to a scene from the Automated Driving Toolbox™ Interface for Unreal Engine® 4 Projects support package, then enter the stencil ID corresponding to that asset type, as shown in the table. If you are adding assets other than the ones shown, then you can assign them to unused IDs. If you do not assign a stencil ID to an asset, then the Unreal Editor assigns that asset an ID of
The Simulation 3D Camera block does not support the output of semantic segmentation data for lane markings. Even if you assign a stencil ID to lane markings, the block ignores this setting.
Generic traffic sign
Speed limit sign
Weight limit sign
Left and right arrow warning sign
Left chevron warning sign
Right chevron warning sign
Right one-way sign
School bus only sign
Curve right warning sign
Curve left warning sign
Up right arrow warning sign
Railroad crossing sign
Roundabout warning sign
Bike lane sign
Road guard rail
For example, for a stop sign that is missing a stencil ID, enter
If you are adding stencil ID for scene elements of the same type, you can copy (Ctrl+C) and paste (Ctrl+V) the element with the added stencil ID. The copied scene element includes the stencil ID.
Visually verify that the correct stencil ID shows by using the custom stencil view. In the top-left corner of the editor window, click and select Buffer Visualization > Custom Stencil. The scene displays the stencil IDs specified for each scene element. For example, if you added the correct stencil ID to a stop sign (
13) then the editor window, the stop sign displays a stencil ID value of
If you did not set a stencil ID value for a scene element, then the element appears in black and displays no stencil ID.
If you did not select CustomDepth Stencil Value, then the scene element does not appear at all in this view.
Turn off the custom stencil ID view. In the top-left corner of the editor window, click Buffer Visualization and then select Lit.
To display semantic segmentation data for your custom labels, follow these steps:
Set up your Simulink model to display semantic segmentation data from a Simulation 3D Camera block. For an example setup, see Depth and Semantic Segmentation Visualization Using Unreal Engine Simulation.
Run the simulation and verify that the Simulation 3D Camera block outputs the correct data. For example, here is the Semantic Segmentation Display window with the correct stencil ID applied to a stop sign.
Simulation 3D Camera | Simulation 3D Scene Configuration | Simulation 3D Vision Detection Generator | Simulation 3D Probabilistic Radar | Simulation 3D Probabilistic Radar Configuration