Automate Real-Time Testing for Forward Vehicle Sensor Fusion

This example uses:

This example shows how to automate testing of a forward vehicle sensor fusion algorithm deployed to a Speedgoat® real-time target machine using Simulink® Test™. In this example, you:

Deploy the forward vehicle sensor fusion algorithm to a Speedgoat machine using Simulink Real-Time™.
Perform automated testing of the deployed application using Simulink Test.

The forward vehicle sensor fusion component performs information fusion from different sensors to perceive front view of the autonomous vehicle. This component is central to the decision-making process in various automated driving applications, such as highway lane following and forward collision warning. This component is commonly deployed to a real-time processor.

This example shows how you can deploy the forward vehicle sensor fusion algorithm to a Speedgoat real-time machine. It also shows how you can reuse the desktop simulation test cases and automate regression testing for the deployed algorithm. This example builds on the Forward Vehicle Sensor Fusion example.

System Configuration

This example uses a hardware setup that primarily consists of two machines, a host and a target, connected by ethernet.

This example uses this hardware configuration:

Target — Speedgoat Performance real-time machine with an Intel® Core™ i7 @ 4.2 GHz, running Simulink Real-Time. For more information, see the Speedgoat Performance Real-Time Target Machine.
Host — Intel® Xeon® @ 3.60GHz, running Windows® 10, 64-bit operating system.
Ethernet cable connecting the target to the host.

The target runs the forward vehicle sensor fusion algorithm. It sends algorithm output to the host using the User Datagram Protocol (UDP) over the ethernet cable.

The host sets up the simulation environment and configures the test scenarios, sensor models, and metrics assessment. It sends vision and radar detection data to the target and receives the tracker output from the target using UDP.

Using this setup, you can deploy a forward vehicle sensor fusion algorithm to the target, run the host model for a test scenario, and log and visualize simulation results.

In this example, you:

Review the simulation test bench model — The simulation test bench model contains the scenario, sensor models, forward vehicle sensor fusion algorithm, and metrics to assess functionality. The metric assessments integrate the test bench model with Simulink Test for automated testing.
Partition and explore the host and target models — The simulation test bench model is partitioned into two models. One runs on the host machine and the other is used for deployment to the target machine.
Deploy the target model — Configure and deploy the forward vehicle sensor fusion algorithm model to the target machine using Simulink Real-Time.
Simulate the host model and visualize the results — Configure the host model with a test scenario. Simulate the model and visualize the results.
Explore the Test Manager file — Explore the configured Test Manager file that enables you to automate the testing of the deployed forward vehicle sensor fusion algorithm.
Automate testing — Run the test suite using the Test Manager and analyze the test report.

Review Simulation Test Bench Model

This example uses the test bench model from the Forward Vehicle Sensor Fusion example.

To explore the test bench model, load the forward vehicle sensor fusion project.

openProject("FVSensorFusion");

Open the test bench model.

open_system("ForwardVehicleSensorFusionTestBench")

This simulation test bench model contains these subsystems:

Sensors and Environment — Specifies the scene, vehicles, and sensors used for simulation.
Forward Vehicle Sensor Fusion — Implements the radar clustering, detection concatenation, fusion, and tracking algorithms.
Evaluate Tracker Metrics — Assesses the tracker performance, using the generalized optimal subpattern assignment (GOSPA) metric, between a set of tracks and their ground truths.

Partition and Explore Host and Target Models

The test bench model is partitioned into host and target models. Explore these models.

Explore Host Model

The host model contains the Sensors and Environment and Evaluate Tracker Metrics subsystems of the test bench model. The model also configures the UDP interface using the Send Sensor Data via UDP and Receive Tracker Data via UDP subsystems. Open the host model.

open_system("RTForwardVehicleSensorFusionHost")

The Sensors and Environment subsystem configures the road network, places vehicles, and synthesizes sensors. For more information, see the Forward Vehicle Sensor Fusion example. Open the Sensors and Environment subsystem.

open_system("RTForwardVehicleSensorFusionHost/Sensors and Environment")

This subsystem also generates the reset flag using a MATLAB Function block. Use this flag to reset the internal states of the deployed application before you run the simulation.

The Send Sensor Data via UDP subsystem contains Byte Packing (Simulink Real-Time) and UDP Send (Simulink Real-Time) blocks from the Simulink Real-Time library. Open the Send Sensor Data via UDP subsystem.

open_system("RTForwardVehicleSensorFusionHost/Send Sensor Data via UDP")

Each Byte Pack subsystem contains a Byte Packing block that converts one or more signals of user-selectable data types to a single vector of varying data types. Each UDP Send block sends the data from the corresponding Byte Pack subsystem over a UDP network to the specified IP address and port. You must configure these UDP Send blocks with the IP address and port number of the target machine.

This example sets the maximum number of vision detections to 20 and maximum number of radar detections to 50. You can update these values in the target setup function helperSLForwardVehicleSensorFusionTargetSetup.

This list defines the specifications of the data signals.

System Time — 8-byte double
Vision Detections — 4825-byte Simulink bus structure
Reset Flag — 8-byte double
Radar Detections — 12809-byte Simulink bus structure

The Receive Tracker Data via UDP subsystem contains UDP Receive (Simulink Real-Time) and Byte Unpacking (Simulink Real-Time) blocks from the Simulink Real-Time library. Open the Receive Tracker Data via UDP subsystem.

open_system("RTForwardVehicleSensorFusionHost/Receive Tracker Data via UDP")

This subsystem contains two additional subsystems:

Receive Tracker Data — Receives tracker output data sent from the target machine.
Unpack Tracks Data — Unpacks the data and convert it into a bus signal.

The Evaluate Tracker Metrics subsystem computes various metrics to assess the performance of a tracker. Open the Evaluate Tracker Metrics subsystem.

open_system("RTForwardVehicleSensorFusionHost/Evaluate Tracker Metrics")

This subsystem implements a GOSPA Metric block to assess the performance of the tracker. For more information, see Generalized Optimal Subpattern Assignment Metric (Sensor Fusion and Tracking Toolbox).

The Evaluate Tracker Metrics subsystem is based on the subsystem used in the Forward Vehicle Sensor Fusion example.

To perform the real-time simulation, the host model runs with simulation pacing set to 0.05. You can tune simulation pacing value to synchronize the host and target machines and reproduce the simulation results shown in this example. For more information on tuning simulation pacing value, see Simulation Pacing Options (Simulink).

Explore Target Model

The target model contains the Forward Vehicle Sensor Fusion subsystem, along with UDP interfaces. Open the target model.

open_system("RTForwardVehicleSensorFusionTarget")

The target model contains these subsystems:

Receive Sensor Data via UDP — Receives data required for the forward vehicle sensor fusion algorithm model to run.
Forward Vehicle Sensor Fusion — Enabled subsystem that contains the forward vehicle sensor fusion algorithm.
Send Tracker Data via UDP — Sends the tracker output to the host model, which is required by Evaluate Tracker Metrics subsystem of the host model.

The Receive Sensor Data via UDP subsystem contains UDP Receive and Byte Unpacking blocks from the Simulink Real-Time library. Open the Receive Sensor Data via UDP subsystem.

open_system("RTForwardVehicleSensorFusionTarget/Receive Sensor Data via UDP")

This subsystem receives data frames from the host machine, and deconstructs them using the Byte Unpack subsystems.

The Send Tracker Data via UDP subsystem contains Byte Packing and UDP Send blocks from the Simulink Real-Time library. Open the Send Tracker Data via UDP subsystem.

open_system("RTForwardVehicleSensorFusionTarget/Send Tracker Data via UDP")

This subsystem sends the tracker output to the host machine using a UDP Send block.

The Forward Vehicle Sensor Fusion subsystem is an enabled subsystem that enables execution of the algorithm upon receiving data from the host machine. Open the Forward Vehicle Sensor Fusion subsystem.

open_system("RTForwardVehicleSensorFusionTarget/Forward Vehicle Sensor Fusion")

The enabled subsystem contains a resettable subsystem. The resettable subsystem resets the forward vehicle sensor fusion algorithm to its default state when Reset Flag is enabled. Open the Resettable Forward Vehicle Sensor Fusion subsystem.

open_system("RTForwardVehicleSensorFusionTarget/Forward Vehicle Sensor Fusion/Resettable Forward Vehicle Sensor Fusion")

This subsystem contains the Forward Vehicle Sensor Fusion reference model. This is the same model used in the Forward Vehicle Sensor Fusion example.

Deploy Target Model

Follow these steps to deploy the model to a real-time machine.

1. Configure the UDP blocks in the host and target models.

The UDP Send and UDP Receive blocks used in the host and target models require valid IP addresses for the host and target machines. This example ships with the helperSLRTUDPSetup.m file, which updates these blocks with your specified IP addresses. You can update these blocks manually, or by using the helperSLRTUDPSetup function as shown:

% Specify host model and IP address of Host machine
hostMdl = "RTForwardVehicleSensorFusionHost";
hostIP = "10.1.10.16";

% Specify Target model and IP address of Target machine
targetMdl = "RTForwardVehicleSensorFusionTarget";
targetIP = "10.1.10.17";

% Invoke the function to update the blocks
helperSLRTUDPSetup(targetMdl,targetIP,hostMdl,hostIP);

2. Connect to the target machine.

Connect to the target machine by defining an slrealtime object to manage the target computer.

% Create slrealtime object
tg = slrealtime;

% Specify IP address for target machine
setipaddr(tg,targetIP)

% Connect to target
connect(tg);

3. Update SystemTargetFile configuration parameter for the target and its referenced models and build the target model, which generates an slrealtime application file, RTForwardVehicleSensorFusionTarget.mldatx.

% Read the system target file name of the target
modelSTF = getSTFName(tg);
% Set SystemTargetFile parameter in the target model
set_param(targetMdl,SystemTargetFile=modelSTF);
save_system(targetMdl);
% Set SystemTargetFile parameter in the ForwardVehicleSensorFusion.slx
% model
load_system("ForwardVehicleSensorFusion");
set_param("ForwardVehicleSensorFusion",SystemTargetFile=modelSTF);
save_system("ForwardVehicleSensorFusion");

% Build the target model
slbuild(targetMdl);

The real-time operating system (RTOS) version on the target computer must match the version on the host computer. Otherwise, you cannot connect to the target computer. Run the update(tg) command to update the RTOS version on the target computer.

4. Load the real-time application to the target.

Load the generated RTForwardVehicleSensorFusionTarget.mldatx application to the target machine.

% Load the generated application to the target
load(tg,targetMdl);

5. Execute the real-time application on the target.

% Start the loaded application on the target machine.
start(tg);

Alternatively, you can deploy the target model by using the Simulink graphical user interface.

On the Real-Time tab, in the Connect To Target Computer section, select your target machine from the list. Use the Simulink Real-Time Explorer (Simulink Real-Time) to configure the target.

To deploy and run the model on the target machine, select Run on Target.

Simulate Host Model and Visualize Results

Configure the RTForwardVehicleSensorFusionHost model to simulate the scenario_LFACC_03_Curve_StopnGo scenario. This scenario contains six vehicles, including the ego vehicle, and defines their trajectories.

helperSLForwardVehicleSensorFusionHostSetup(scenarioFcnName="scenario_LFACC_03_Curve_StopnGo");

Simulate the host model.

sim(hostMdl);

Stop and unload the application in the target machine.

stop(tg);

Plot the performance metrics for the target algorithm. For more information about these performance metrics, see Forward Vehicle Sensor Fusion example.

hFigure = helperPlotForwardVehicleSensorFusionResults(logsout);

The plots show that the localization error is the primary factor in the GOSPA metric. Notice that the missed target component initially starts from a higher value due to the establishment delay of the tracker and goes down to zero after some time. The other peaks in the missed target curve occur because of the same delay, when the yellow and purple target vehicles enter the coverage area of the sensors.

Close the figure.

close(hFigure);

Explore Test Manager File

Simulink Test includes the Test Manager, which you can use to author test cases for Simulink models. After authoring your test cases, you can group them and execute them individually or in a batch.

This example reuses the test scenarios from the Automate Testing for Forward Vehicle Sensor Fusion example.

The target model is configured so that you do not have to compile and deploy the target model for every test case. Once you deploy the model, and before the next test case, it resets to its initial state. The example uses these test file callbacks:

The SETUP callback calls the helperRTTestManagerSetup function to configure the target model to deploy and run on a real-time machine.
The CLEANUP callback stops and unloads the application from the target machine.

Open the test file RTForwardVehicleSensorFusionTests.mldatx in the Test Manager and explore the configuration.

sltestmgr
sltest.testmanager.load("RTForwardVehicleSensorFusionTests.mldatx");

You can also run individual test scenarios using these test case callbacks:

The POST-LOAD callback calls the helperRTTestManagerSetup to configure the target model to deploy and run on a real-time machine. It also configures the host model for a test scenario.
The CLEANUP callback contains a script to plot the results of the simulation run.

Automate Testing

Using the created Test Manager file, run a single test case to test the application on a real-time machine.

Running the test scenarios from test file requires a connection with the target machine. If the target machine is not connected, refer to steps 1–3 of the Deploy Target Model section to establish the connection.

Use this code to test the forward vehicle sensor fusion algorithm model with the scenario_LFACC_03_Curve_StopnGo test scenario from Simulink Test.

testFile = sltest.testmanager.load("RTForwardVehicleSensorFusionTests.mldatx");
testSuite = getTestSuiteByName(testFile,"Test Scenarios");
testCase = getTestCaseByName(testSuite,"scenario_LFACC_03_Curve_StopnGo");
resultObj = run(testCase);

To generate a report after the simulation, use this code:

sltest.testmanager.report(resultObj,"Report.pdf", ...
Title="Real-Time Forward Vehicle Sensor Fusion", ...
IncludeMATLABFigures=true, ...
IncludeErrorMessages=true, ...
IncludeTestResults=false, ...
LaunchReport=true);

Examine Report.pdf. Observe that the Test Environment section shows the platform on which the test is run and the MATLAB version used for testing. The Summary section shows the outcome of the test and the duration of the simulation in seconds. The Results section shows pass or fail results based on the assessment criteria. This section also shows the logged plots from the CLEANUP callback commands.

Run and Explore Results for All Test Scenarios

You can simulate the system for all the tests by using run(testFile). Alternatively, you can simulate the system by clicking Run in the Test Manager.

When the test simulations are complete, you can view the results for all the tests in the Results and Artifacts pane of the Test Manager. For each test case, the Check Static Range (Simulink) blocks in the model are associated with the Test Manager to visualize overall pass or fail results.

You can find the generated report in the current working directory. This report contains a detailed summary of the pass or fail statuses and plots for each test case.