Integrate Lidar Sensor Model into RoadRunner Scenario and Cosimulate with Unreal Engine

Since R2024a

This example uses:

This example shows how to integrate a lidar sensor model into RoadRunner Scenario using Simulink® and cosimulate the scenario with Unreal Engine® from Epic Games®.

Introduction

The Lidar Sensor block enables you to simulate high-fidelity point cloud data. You can configure the lidar sensor model to simulate scan patterns of different popular sensors, such as the Velodyne® HDL32E and Ouster® OS0-128. You can also simulate the effects of motion distortion and weather conditions. For more information, see Add Lidar Sensor Model with Simulated Weather Effects to RoadRunner Scenario.

In this example, you configure the lidar sensor model to simulate point cloud data from a Velodyne HDL32E sensor and integrate the sensor into RoadRunner Scenario, which is an interactive editor that enables you to design scenarios for simulating and testing automated driving systems. You define the sensor model in Simulink, and then add it to the vehicle actors in a RoadRunner scenario. Then, obtain generated point cloud data from RoadRunner Scenario, and visualize it in Simulink. You can also cosimulate the scenario in Unreal Engine from Epic Games, which requires a Windows® 64-bit platform.

if ~ispc
    error("Unreal Engine simulation is supported only on Microsoft"+char(174)+" Windows"+char(174)+".")
end

This example also requires you to download the Automated Driving Toolbox™ Interface for Unreal Engine Projects support package. For more information on downloading and installing support packages, see Get and Manage Add-Ons.

pathToAssets = fullfile(matlabshared.supportpkg.getSupportPackageRoot, ...
    "toolbox","shared","sim3dprojects","spkg","roadrunner", ...
    "RoadRunnerProject","Assets");

if (~exist(pathToAssets,"file"))
    error("This example requires you to download and install Automated Driving Toolbox Interface for Unreal Engine(R) Projects support package.")
end

Set Up RoadRunner Environment

Set up an environment for cosimulating a RoadRunner scenario with MATLAB®. Specify the path to your local RoadRunner installation folder. This code shows the path for the default installation location on Windows.

rrInstallationPath = "C:\Program Files\RoadRunner R2023b\bin\win64";

Specify the path to your RoadRunner project. This code shows the path to a sample project folder on Windows.

rrProjectPath = "C:\RR\MyProjects";

Update the path for the RoadRunner installation folder by first getting the root object within the settings hierarchical tree. For more information, see SettingsGroup.

s = settings;
s.roadrunner.application.InstallationFolder.PersonalValue = rrInstallationPath;

Open RoadRunner using the specified path to your project. The roadrunner object rrApp enables you to interact with RoadRunner from the MATLAB workspace. You can open a RoadRunner scenario using this object.

rrApp = roadrunner(rrProjectPath);

This example uses these files, which you must add to the RoadRunner project.

SimpleScene.rrscene — RoadRunner scene file that describes a simple RoadRunner scene. This scene is based on the ScenarioBasic.rrscene scene included with RoadRunner. It contains a four-way intersection with traffic signals, a roundabout, and several roads of varying lengths and curve types.
SimpleScenario.rrscenario — RoadRunner scenario file that specifies two actors and defines their trajectories.
RRLidarSensorEgoBehavior.rrbehavior.rrmeta — Behavior file that associates the sensor detection processing and visualization behavior implemented using the Simulink model to the ego vehicle in the RoadRunner scenario.

Copy these files to the RoadRunner project. For more information on the RoadRunner environment, see RoadRunner Project and Scene System (RoadRunner).

copyfile("SimpleScene.rrscene",fullfile(rrProjectPath,"Scenes/"))
copyfile("SimpleScenario.rrscenario",fullfile(rrProjectPath,"Scenarios/"))
copyfile("RRLidarSensorEgoBehavior.rrbehavior.rrmeta",fullfile(rrProjectPath,"Assets","Behaviors/"))

This example shows you how to perform cosimulation of a RoadRunner scenario with Unreal Engine from Epic Games. To integrate a RoadRunner scene into Unreal Engine, open the scene in RoadRunner and export the scene to a FBX file.

if ~exist(fullfile(rrProjectPath,"Exports/result/SimpleScene.fbx"),"file")
    options = unrealExportOptions(FilmboxOptions=filmboxExportOptions(EmbedTextures=true,ExportOnlyHighestLOD=true));
    openScene(rrApp,"SimpleScene");
    exportScene(rrApp,"result/SimpleScene.fbx","Unreal",options);
end

Explore RoadRunner Scenario

Open the SimpleScenario.rrscenario scenario.

openScenario(rrApp,"SimpleScenario")

The scenario contains two vehicles. The white lead car follows the lane-following built-in behavior. The yellow ego car follows the custom path-planner behavior using the RRLidarSensorEgoBehavior.rrbehavior.rrmeta file, which points to the rrScenarioSimWithLidarSensor.slx Simulink model.

Connect to the RoadRunner Scenario server for cosimulation by using the createSimulation function.

rrSim = createSimulation(rrApp);

Specify the step size for the RoadRunner scenario simulation. Units are in seconds.

simStepSize = 0.1;
set(rrSim,StepSize=simStepSize);

Explore Simulink Model

The Simulink model rrScenarioSimWithLidarSensor contains interfaces for communicating with RoadRunner Scenario and Unreal Engine. The model specifies a lidar sensor and camera sensor for the ego vehicle. The model also reads the path defined for the ego vehicle from RoadRunner Scenario, implements path-follower logic, and controls ego vehicle motion along the path. Load the bus definitions and open the model.

load("busActorsForSimLSM.mat")
modelName = 'rrScenarioSimWithLidarSensor';
open_system(modelName)

RoadRunner Scenario Blocks

The model contains these RoadRunner Scenario blocks:

RoadRunner Scenario (Automated Driving Toolbox) — Defines the interface for an actor model.
Self Actor Runtime Reader — RoadRunner Scenario Reader (Automated Driving Toolbox) block that reads ego actor runtime information.
All Actor Runtime — RoadRunner Scenario Reader block that reads the runtime information of all the vehicles.
Path Action Reader — RoadRunner Scenario Reader block that reads the reference path of the ego vehicle.
Self Actor Runtime Writer — RoadRunner Scenario Writer (Automated Driving Toolbox) block that writes the ego vehicle runtime to RoadRunner Scenario.
Path Follower — MATLAB Function block that implements the path-follower logic and controls the ego vehicle motion along the path.

3D Simulation Blocks

Simulation 3D Scene Configuration (Automated Driving Toolbox) — Specifies the scene for Unreal Engine simulation. For this example, the Scene source parameter value of the block is RoadRunner, and the Project parameter specifies the path to an FBX file.
Get Actor 3D Simulation Coordinates — Converts the driving-scenario-compatible pose from RoadRunner Scenario into the format required by the 3D simulation environment.
Simulation 3D Vehicle with Ground Following (Automated Driving Toolbox) — Implements a vehicle at the specified position in the 3D simulation environment. This example uses two such blocks: one for the ego vehicle and another for the target vehicle.
Simulation 3D Camera (Automated Driving Toolbox) — Captures the front view of the ego vehicle in the 3D simulation environment. In this example, the value of the Parent name parameter of the block is EgoCar.
Video Viewer — Visualizes camera images generated by the Simulation 3D Camera block.

Lidar Sensor and Point Cloud Visualization Blocks

Lidar Sensor — Simulates a high-fidelity lidar sensor, and generates point cloud data.
Visualize Point Cloud — Displays generated point cloud data using the pcplayer object.

To configure the scene for Unreal Engine simulation, set the Project parameter of the Simulation 3D Scene Configuration block to the exported SimpleScene.fbx file.

rrFBXPath = fullfile(rrProjectPath,"Exports/result/SimpleScene.fbx");
set_param(modelName+"/Simulation 3D Scene Configuration",RoadRunnerProjectPath=rrFBXPath);

To control the sensor configuration from the RoadRunner Scenario simulation object, retrieve a SensorSimulation (Automated Driving Toolbox) object. Then, specify the lidar sensor block path in the model.

sensorSim = get(rrSim,"SensorSimulation");
lidarSensorBlkPath = [modelName,'/Lidar Sensor'];
egoVehicleId = 1;

Use the addSensors (Automated Driving Toolbox) object function of the SensorSimulation object to attach the lidar sensor to the ego vehicle.

addSensors(sensorSim,lidarSensorBlkPath,egoVehicleId);

The model might take a few seconds to update the diagram when compiling for the first time. Update the model before running the simulation.

set_param(modelName,SimulationCommand="Update")

Simulate Scenario

Start the scenario simulation. The model visualizes the 3D point cloud data from RoadRunner Scenario and the camera image from the Simulation 3D Camera during the simulation.

set(rrSim,SimulationCommand="Start")

while ~isequal(get(rrSim,"SimulationStatus"),"Stopped")
    pause(0.5)
end