Verify an Airborne Deep Learning System
This example shows how to verify a deep learning system for airborne applications. You explore a case study of a runway sign classification (RSC) system that receives images from a forward-facing camera, and then detects airport runway signs in those images using object detection networks. This figure summarizes the system.
You compare the performance of each network and, for the network with the best performance, you generate code for CPU and GPU targets. Finally, you compare the performance and inference speed of the original network and the generated MEX files.
This example is based on the work in [5], which includes the development and verification activities required by DO-178C [1], ARP4754A [2], and prospective EASA and FAA guidelines [3,4]. In this example, you carry out deep-learning-specific activities that support certification against these standards. To verify that the system complies with the full aviation industry standards and prospective guidelines, see Runway Sign Classifier: Certify an Airborne Deep Learning System (DO Qualification Kit).
This example consists of a project with four folders. The Data
, Implementation
, and Learning
folders each contain live scripts. To run this example successfully, you must run the live scripts in this order.
Data Management
Data Traceability
Data Analysis
Data Reviews
Data Allocation
Data Preparation
Learning Management
Estimate Anchor Boxes
Train YOLO v2 Network
Train YOLO v3 Network
Train YOLO v4 Network
Model Evaluation
Model Implementation
Generate Code on CPU
Quantize Model and Generate Code on GPU
Implementation Evaluation
Data Management
Data sets for deep neural network (DNN) training, validation, and testing are specific to your artificial intelligence (AI) system. Following the assumptions in [6], you treat DNN training and validation data set requirements as equivalent to software requirements that specify the behavior of a DNN model. For information about data development activities, including collection, labeling, and traceability, see Data Management.
Because training and validation data sets are subject to software requirements, you must verify their accuracy, consistency, traceability, and compliance in accordance with DO-178C.
The Data Management live script also shows how to perform these tasks:
Verify data consistency by using the data validity analysis methods.
Verify data accuracy by using the Image Labeler app.
Learning Management and Network Training
Learning management includes training preparation activities such as specifying the model architecture, training algorithm, and initial hyperparameters estimate. An AI model that you produce as a result of network training represents the software design from which you generate software source code.
For more information about learning management and model training activities, see Learning Management. For information about model testing as an essential part of the AI workflow, see Model Testing and Selection in Learning Management.
Model Implementation
Use model implementation activities to produce source code and build executable code that runs on host hardware for evaluation and is ready for deployment on embedded hardware.
For information about code generation and related model optimization activities such as quantization, see Model Implementation. Verifying the model implementation involves testing the executable code. This example reuses the model testing data set to test the implementation. For more information, see Implementation Evaluation in Model Implementation.
References
[1] RTCA DO-178C. "Software Considerations in Airborne Systems and Equipment Certification." RTCA SC-205, EUROCAE WG-12. https://www.rtca.org/.
[2] SAE ARP4754A. "Guidelines for Development of Civil Aircraft and Systems." SAE International. https://www.sae.org.
[3] Soudain, Guillaume. "EASA Concept Paper: First Usable Guidance for Level 1 Machine Learning Applications." Technical report, European Aviation Safety Agency, 2021. https://www.easa.europa.eu/en.
[4] Balduzzi, Giovanni, Martino Ferrari Bravo, Anna Chernova, Calin Cruceru, Luuk van Dijk, Peter de Lange, Juan Jerez, et al. "Neural Network Based Runway Landing Guidance for General Aviation Autoland." Technical report DOT/FAA/TC-21/48, Federal Aviation Administration, 2021.
[5] Dmitriev, Konstantin, Johann Schumann, and Florian Holzapfel. “Toward Certification of Machine-Learning Systems for Low Criticality Airborne Applications.” In 2021 IEEE/AIAA 40th Digital Avionics Systems Conference (DASC), 1–7. San Antonio, TX, USA: IEEE, 2021. https://doi.org/10.1109/DASC52595.2021.9594467.
[6] Dmitriev, Konstantin, Johann Schumann, and Florian Holzapfel. “Toward Design Assurance of Machine-Learning Airborne Systems.” In AIAA SCITECH 2022 Forum. San Diego, CA & Virtual: American Institute of Aeronautics and Astronautics, 2022. https://doi.org/10.2514/6.2022-1134.
See Also
Apps
- Deep Network Designer | Deep Network Quantizer | Image Labeler (Computer Vision Toolbox)
Functions
codegen
(MATLAB Coder) |estimateAnchorBoxes
(Computer Vision Toolbox) |yolov2Layers
(Computer Vision Toolbox)
Objects
boxLabelDatastore
(Computer Vision Toolbox) |dlquantizer
|groundTruth
(Computer Vision Toolbox) |minibatchqueue
|mobilenetv2
|squeezenet
|yolov3ObjectDetector
(Computer Vision Toolbox) |yolov4ObjectDetector
(Computer Vision Toolbox)
Related Topics
- Verify Robustness of Deep Learning Neural Network
- Train Image Classification Network Robust to Adversarial Examples
- Out-of-Distribution Detection for Deep Neural Networks
- Out-of-Distribution Data Discriminator for YOLO v4 Object Detector
- Runway Sign Classifier: Certify an Airborne Deep Learning System (DO Qualification Kit)