Automate Real-Time Testing for Forward Vehicle Sensor Fusion
This example shows how to automate testing of a forward vehicle sensor fusion algorithm deployed to a Speedgoat® real-time target machine using Simulink® Test™. In this example, you:
Deploy the forward vehicle sensor fusion algorithm to a Speedgoat machine using Simulink Real-Time™.
Perform automated testing of the deployed application using Simulink Test.
The forward vehicle sensor fusion component performs information fusion from different sensors to perceive front view of the autonomous vehicle. This component is central to the decision-making process in various automated driving applications, such as highway lane following and forward collision warning. This component is commonly deployed to a real-time processor.
This example shows how you can deploy the forward vehicle sensor fusion algorithm to a Speedgoat real-time machine. It also shows how you can reuse the desktop simulation test cases and automate regression testing for the deployed algorithm. This example builds on the Forward Vehicle Sensor Fusion example.
This example uses a hardware setup that primarily consists of two machines, a host and a target, connected by ethernet.
This example uses this hardware configuration:
Target — Speedgoat Performance real-time machine with an Intel® Core™ i7 @ 4.2 GHz, running Simulink Real-Time. For more information, see the Speedgoat Performance real-time target machine site.
Host — Intel® Xeon® @ 3.60GHz, running Windows® 10, 64-bit operating system.
Ethernet cable connecting the target to the host.
The target runs the forward vehicle sensor fusion algorithm. It sends algorithm output to the host using the User Datagram Protocol (UDP) over the ethernet cable.
The host sets up the simulation environment and configures the test scenarios, sensor models, and metrics assessment. It sends vision and radar detection data to the target and receives the tracker output from the target using UDP.
Using this setup, you can deploy a forward vehicle sensor fusion algorithm to the target, run the host model for a test scenario, and log and visualize simulation results.
In this example, you:
Review the simulation test bench model — The simulation test bench model contains the scenario, sensor models, forward vehicle sensor fusion algorithm, and metrics to assess functionality. The metric assessments integrate the test bench model with Simulink Test for automated testing.
Partition and explore the host and target models — The simulation test bench model is partitioned into two models. One runs on the host machine and the other is used for deployment to the target machine.
Deploy the target model — Configure and deploy the forward vehicle sensor fusion algorithm model to the target machine using Simulink Real-Time.
Simulate the host model and visualize the results — Configure the host model with a test scenario. Simulate the model and visualize the results.
Explore the Test Manager file — Explore the configured
Test Managerfile that enables you to automate the testing of the deployed forward vehicle sensor fusion algorithm.
Automate testing — Run the test suite using the Test Manager and analyze the test report.
Review Simulation Test Bench Model
This example uses the test bench model from the Forward Vehicle Sensor Fusion example.
To explore the test bench model, open a working copy of the project example files. MATLAB® copies the files to an example folder so you can edit them.
Open the test bench model.
This simulation test bench model contains these subsystems:
Sensors and Environment— Specifies the scene, vehicles, and sensors used for simulation.
Forward Vehicle Sensor Fusion— Implements the radar clustering, detection concatenation, fusion, and tracking algorithms.
Evaluate Tracker Metrics— Assesses the tracker performance, using the generalized optimal subpattern assignment (GOSPA) metric, between a set of tracks and their ground truths.
Partition and Explore Host and Target Models
The test bench model is partitioned into host and target models. Explore these models.
Explore Host Model
The host model contains the
Sensors and Environment and
Evaluate Tracker Metrics subsystems of the test bench model. The model also configures the UDP interface using the
Send Sensor Data via UDP and
Receive Tracker Data via UDP subsystems. Open the host model.
Sensors and Environment subsystem configures the road network, places vehicles, and synthesizes sensors. For more information, see the Forward Vehicle Sensor Fusion example. Open the
Sensors and Environment subsystem.
open_system("RTForwardVehicleSensorFusionHost/Sensors and Environment")
This subsystem also generates the reset flag using a MATLAB Function block. Use this flag to reset the internal states of the deployed application before you run the simulation.
Send Sensor Data via UDP subsystem contains Byte Packing (Simulink Real-Time) and UDP Send (Simulink Real-Time) blocks from the Simulink Real-Time library. Open the
Send Sensor Data via UDP subsystem.
open_system("RTForwardVehicleSensorFusionHost/Send Sensor Data via UDP")
Byte Pack subsystem contains a Byte Packing block that converts one or more signals of user-selectable data types to a single vector of varying data types. Each UDP Send block sends the data from the corresponding Byte Pack subsystem over a UDP network to the specified IP address and port. You must configure these UDP Send blocks with the IP address and port number of the target machine.
This example sets the maximum number of vision detections to 20 and maximum number of radar detections to 50. You can update these values in the target setup function
This list defines the specifications of the data signals.
System Time— 8-byte
Vision Detections— 4825-byte Simulink bus structure
Reset Flag— 8-byte
Radar Detections— 12809-byte Simulink bus structure
Receive Tracker Data via UDP subsystem contains UDP Receive (Simulink Real-Time) and Byte Unpacking (Simulink Real-Time) blocks from the Simulink Real-Time library. Open the
Receive Tracker Data via UDP subsystem.
open_system("RTForwardVehicleSensorFusionHost/Receive Tracker Data via UDP")
This subsystem contains two additional subsystems:
Receive Tracker Data— Receives tracker output data sent from the target machine.
Unpack Tracks Data— Unpacks the data and convert it into a bus signal.
Evaluate Tracker Metrics subsystem computes various metrics to assess the performance of a tracker. Open the
Evaluate Tracker Metrics subsystem.
open_system("RTForwardVehicleSensorFusionHost/Evaluate Tracker Metrics")
This subsystem implements a GOSPA Metric block to assess the performance of the tracker. For more information, see Generalized Optimal Subpattern Assignment Metric (Sensor Fusion and Tracking Toolbox).
Evaluate Tracker Metrics subsystem is based on the subsystem used in the Forward Vehicle Sensor Fusion example.
To perform the real-time simulation, the host model runs with simulation pacing set to
0.05. You can tune simulation pacing value to synchronize the host and target machines and reproduce the simulation results shown in this example. For more information on tuning simulation pacing value, see Simulation Pacing (Simulink).
Explore Target Model
The target model contains the
Forward Vehicle Sensor Fusion subsystem, along with UDP interfaces. Open the target model.
The target model contains these subsystems:
Receive Sensor Data via UDP— Receives data required for the forward vehicle sensor fusion algorithm model to run.
Forward Vehicle Sensor Fusion— Enabled subsystem that contains the forward vehicle sensor fusion algorithm.
Send Tracker Data via UDP— Sends the tracker output to the host model, which is required by
Evaluate Tracker Metricssubsystem of the host model.
Receive Sensor Data via UDP subsystem contains UDP Receive and Byte Unpacking blocks from the Simulink Real-Time library. Open the
Receive Sensor Data via UDP subsystem.
open_system("RTForwardVehicleSensorFusionTarget/Receive Sensor Data via UDP")
This subsystem receives data frames from the host machine, and deconstructs them using the
Byte Unpack subsystems.
Send Tracker Data via UDP subsystem contains Byte Packing and UDP Send blocks from the Simulink Real-Time library. Open the
Send Tracker Data via UDP subsystem.
open_system("RTForwardVehicleSensorFusionTarget/Send Tracker Data via UDP")
This subsystem sends the tracker output to the host machine using a UDP Send block.
Forward Vehicle Sensor Fusion subsystem is an enabled subsystem that enables execution of the algorithm upon receiving data from the host machine. Open the
Forward Vehicle Sensor Fusion subsystem.
open_system("RTForwardVehicleSensorFusionTarget/Forward Vehicle Sensor Fusion")
The enabled subsystem contains a resettable subsystem. The resettable subsystem resets the forward vehicle sensor fusion algorithm to its default state when
Reset Flag is enabled. Open the
Resettable Forward Vehicle Sensor Fusion subsystem.
open_system("RTForwardVehicleSensorFusionTarget/Forward Vehicle Sensor Fusion/Resettable Forward Vehicle Sensor Fusion")
This subsystem contains the
Forward Vehicle Sensor Fusion reference model. This is the same model used in the Forward Vehicle Sensor Fusion example.
Deploy Target Model
Follow these steps to deploy the model to a real-time machine.
1. Configure the UDP blocks in the host and target models.
The UDP Send and UDP Receive blocks used in the host and target models require valid IP addresses for the host and target machines. This example ships with the
helperSLRTUDPSetup.m file, which updates these blocks with your specified IP addresses. You can update these blocks manually, or by using the
helperSLRTUDPSetup function as shown:
% Specify host model and IP address of Host machine hostMdl = "RTForwardVehicleSensorFusionHost"; hostIP = "10.1.10.16";
% Specify Target model and IP address of Target machine targetMdl = "RTForwardVehicleSensorFusionTarget"; targetIP = "10.1.10.17";
% Invoke the function to update the blocks helperSLRTUDPSetup(targetMdl,targetIP,hostMdl,hostIP);
2. Set up real-time configuration to the reference model
ForwardVehicleSensorFusion.slx, and build the target model, which creates an
slrealtime application file,
% Setup real-time configuration to the ForwardVehicleSensorFusion.slx load_system("ForwardVehicleSensorFusion"); if (isempty(getConfigSet("ForwardVehicleSensorFusion","ConfigRef"))) configRef = Simulink.ConfigSetRef; attachConfigSet("ForwardVehicleSensorFusion",configRef) set_param(configRef,SourceName="RTConfig"); set_param(configRef,Name="ConfigRef"); setActiveConfigSet("ForwardVehicleSensorFusion","ConfigRef"); end save_system('ForwardVehicleSensorFusion');
% Build the target model slbuild(targetMdl);
3. Connect to the target machine.
Connect to the target machine by defining an
slrealtime object to manage the target computer.
% Create slrealtime object tg = slrealtime;
% Specify IP address for target machine setipaddr(tg,targetIP)
% Connect to target connect(tg);
The real-time operating system (RTOS) version on the target computer must match the version on the host computer. Otherwise, you cannot connect to the target computer. Run the
update(tg) command to update the RTOS version on the target computer.
4. Load the real-time application to the target.
Load the generated
RTForwardVehicleSensorFusionTarget.mldatx application to the target machine.
% Load the generated application to the target load(tg,targetMdl);
5. Execute the real-time application on the target.
% Start the loaded application on the target machine. start(tg);
Alternatively, you can deploy the target model by using the Simulink graphical user interface.
On the Real-Time tab, in the Connect To Target Computer section, select your target machine from the list. Use the Simulink Real-Time Explorer (Simulink Real-Time) to configure the target.
To deploy and run the model on the target machine, select Run on Target.
Simulate Host Model and Visualize Results
RTForwardVehicleSensorFusionHost model to simulate the
scenario_LFACC_03_Curve_StopnGo scenario. This scenario contains six vehicles, including the ego vehicle, and defines their trajectories.
Simulate the host model.
Stop and unload the application in the target machine.
Plot the performance metrics for the target algorithm. For more information about these performance metrics, see Forward Vehicle Sensor Fusion example.
hFigure = helperPlotForwardVehicleSensorFusionResults(logsout);
The plots show that the localization error is the primary factor in the GOSPA metric. Notice that the missed target component initially starts from a higher value due to the establishment delay of the tracker and goes down to zero after some time. The other peaks in the missed target curve occur because of the same delay, when the yellow and purple target vehicles enter the coverage area of the sensors.
Close the figure.
Explore Test Manager File
Simulink Test includes the Test Manager, which you can use to author test cases for Simulink models. After authoring your test cases, you can group them and execute them individually or in a batch.
This example reuses the test scenarios from the Automate Testing for Forward Vehicle Sensor Fusion example.
The target model is configured so that you do not have to compile and deploy the target model for every test case. Once you deploy the model, and before the next test case, it resets to its initial state. The example uses these test file callbacks:
The SETUP callback calls the
helperRTTestManagerSetupfunction to configure the target model to deploy and run on a real-time machine.
The CLEANUP callback stops and unloads the application from the target machine.
Open the test file
RTForwardVehicleSensorFusionTests.mldatx in the Test Manager and explore the configuration.
You can also run individual test scenarios using these test case callbacks:
The POST-LOAD callback calls the
helperRTTestManagerSetupto configure the target model to deploy and run on a real-time machine. It also configures the host model for a test scenario.
The CLEANUP callback contains a script to plot the results of the simulation run.
Using the created Test Manager file, run a single test case to test the application on a real-time machine.
Running the test scenarios from test file requires a connection with the target machine. If the target machine is not connected, refer to steps 1–3 of the Deploy Target Model section to establish the connection.
Use this code to test the forward vehicle sensor fusion algorithm model with the
scenario_LFACC_03_Curve_StopnGo test scenario from Simulink Test.
testFile = sltest.testmanager.load("RTForwardVehicleSensorFusionTests.mldatx"); testSuite = getTestSuiteByName(testFile,"Test Scenarios"); testCase = getTestCaseByName(testSuite,"scenario_LFACC_03_Curve_StopnGo"); resultObj = run(testCase);
To generate a report after the simulation, use this code:
sltest.testmanager.report(resultObj,"Report.pdf", ... Title="Real-Time Forward Vehicle Sensor Fusion", ... IncludeMATLABFigures=true, ... IncludeErrorMessages=true, ... IncludeTestResults=false, ... LaunchReport=true);
Report.pdf. Observe that the
Test Environment section shows the platform on which the test is run and the MATLAB version used for testing. The
Summary section shows the outcome of the test and the duration of the simulation in seconds. The
Results section shows pass or fail results based on the assessment criteria. This section also shows the logged plots from the CLEANUP callback commands.
Run and Explore Results for All Test Scenarios
You can simulate the system for all the tests by using
run(testFile). Alternatively, you can simulate the system by clicking Run in the Test Manager.
When the test simulations are complete, you can view the results for all the tests in the Results and Artifacts pane of the Test Manager. For each test case, the Check Static Range (Simulink) blocks in the model are associated with the Test Manager to visualize overall pass or fail results.
You can find the generated report in the current working directory. This report contains a detailed summary of the pass or fail statuses and plots for each test case.
- Scenario Reader | Simulation 3D Scene Configuration | UDP Send (Simulink Real-Time) | UDP Receive (Simulink Real-Time) | Byte Unpacking (Simulink Real-Time) | Byte Packing (Simulink Real-Time)