aucMetric

Deep learning area under ROC curve (AUC) metric

Since R2023b

Description

Use an AUCMetric object to track the area under ROC curve (AUC) value when you train a deep neural network.

To specify which metrics to use during training, specify the Metrics option of the trainingOptions function. You can use this option only when you train a network using the trainnet function.

To plot the metrics during training, in the training options, specify Plots as "training-progress". If you specify the ValidationData training option, then the software also plots and records the metric values for the validation data. To output the metric values to the Command Window during training, in the training options, set Verbose to true.

You can also access the metrics after training using the TrainingHistory and ValidationHistory fields from the second output of the trainnet function.

Creation

Syntax

metric = aucMetric

metric = aucMetric(Name=Value)

Description

metric = aucMetric creates an AUCMetric object. You can then specify metric as the Metrics name-value argument in the trainingOptions function. This metric is valid only for classification tasks.

With no additional options specified, this syntax is equivalent to setting Metrics="auc" in the training options.

metric = aucMetric(Name=Value) sets the Name, NetworkOutput, and AverageType properties using name-value arguments.

Properties

Name — Metric name

"AUC" (default) | string scalar | character vector

Metric name, specified as a string scalar or character vector. The metric name appears in the training plot, the verbose output, and the training information that you can access as the second output of the trainnet function.

Data Types: char | string

NetworkOutput — Name of layer to apply metric to

[] (default) | string scalar | character vector

This property is read-only.

Name of the layer to apply the metric to, specified as [], a string scalar, or a character vector. When the value is [], the software passes all of the network outputs to the metric.

Note

You can apply the built-in metric to only a single output. If you have a network with multiple outputs, then you must specify the NetworkOutput name-value argument. To apply built-in metrics to multiple outputs, you must create a metric object for each output.

Data Types: char | string

AverageType — Type of averaging to use

"micro" (default) | "macro" | "weighted"

This property is read-only.

Type of averaging to use to compute the metric, specified as one of these values:

• "micro" — Calculate the metric across all classes.

• "macro" — Calculate the metric for each class and return the average.

• "weighted" — Calculate the metric for each class and return the weighted average. The weight for a class is the proportion of observations from that class.

For more information, see Averaging Type.

Data Types: char | string

Maximize — Flag to maximize metric

1 (true)

This property is read-only.

Flag to maximize metric, specified as 1 (true) if the optimal value for the metric occurs when the metric is maximized.

For this metric, the Maximize value is always set to 1 (true).

Data Types: single | double | int8 | int16 | int32 | int64 | uint8 | uint16 | uint32 | uint64

Object Functions

trainingOptions	Options for training deep learning neural network
trainnet	Train deep learning neural network

Examples

Plot and Record AUC During Training

Plot and record the training and validation area under the ROC curve (AUC) value when you train a deep neural network.

Unzip the digit sample data and create an image datastore. The imageDatastore function automatically labels the images based on folder names.

unzip("DigitsData.zip")
imds = imageDatastore("DigitsData", ...
    IncludeSubfolders=true, ...
    LabelSource="foldernames");

The datastore contains 10,000 synthetic images of digits from 0 to 9. Each image in the data set has a size of 28-by-28-by-1 pixels. You can train a deep learning network to classify the digit in the image.

Use a subset of the data as the validation set.

numTrainingFiles = 750;
[imdsTrain,imdsVal] = splitEachLabel(imds,numTrainingFiles,"randomize");

Create an image classification network.

layers = [ ...
    imageInputLayer([28 28 1])
    convolution2dLayer(5,20)
    reluLayer
    maxPooling2dLayer(2,Stride=2)
    fullyConnectedLayer(10)
    softmaxLayer];

Create an AUCMetric object and set AverageType to "macro". You can use this object to record and plot the training and validation AUC.

metric = aucMetric(AverageType="macro")

metric = 
  AUCMetric with properties:

             Name: "AUC"
      AverageType: "macro"
    NetworkOutput: []
         Maximize: 1

Specify the AUC metric in the training options. To plot the AUC during training, set Plots to "training-progress". To output the values during training, set Verbose to true.

options = trainingOptions("adam", ...
    MaxEpochs=5, ...
    Metrics=metric, ...
    ValidationData=imdsVal, ...
    ValidationFrequency=50, ...
    Plots="training-progress", ...
    Verbose=true);

Train the network using the trainnet function.

[net,info] = trainnet(imdsTrain,layers,"crossentropy",options);

    Iteration    Epoch    TimeElapsed    LearnRate    TrainingLoss    ValidationLoss    TrainingAUC    ValidationAUC
    _________    _____    ___________    _________    ____________    ______________    ___________    _____________
            0        0       00:00:14        0.001                            13.488                         0.50008
            1        1       00:00:14        0.001          13.974                          0.47203                 
           50        1       00:00:50        0.001          2.7423            2.7443         0.8837          0.88022
          100        2       00:01:21        0.001          1.2964              1.22        0.94237          0.94923
          150        3       00:01:50        0.001         0.65054           0.80056         0.9809          0.96651
          200        4       00:02:17        0.001         0.19025           0.53178        0.99771          0.98268
          250        5       00:02:53        0.001         0.15676           0.49591        0.99603          0.98431
          290        5       00:03:15        0.001         0.27693           0.40591        0.99295          0.98808
Training stopped: Max epochs completed

Access the loss and AUC values for the validation data.

info.ValidationHistory

ans=7×3 table
    Iteration     Loss        AUC  
    _________    _______    _______

         0        13.488    0.50008
        50        2.7443    0.88022
       100          1.22    0.94923
       150       0.80056    0.96651
       200       0.53178    0.98268
       250       0.49591    0.98431
       290       0.40591    0.98808

More About

Area Under ROC Curve (AUC)

A ROC curve shows the true positive rate (TPR), or sensitivity, versus the false positive rate (FPR), or 1-specificity, for different thresholds of classification scores. The area under a ROC curve (AUC) corresponds to the integral of the curve (TPR values) with respect to FPR values from zero to one. The AUC provides an aggregate performance measure across all possible thresholds. The AUC values are in the range [0, 1]. Larger AUC values indicate better classifier performance.

• A perfect classifier always correctly assigns positive class observations to the positive class and has a TPR of 1 for all threshold values corresponding to an AUC value of 1.

• A random classifier returns random score values and has the same FPR and TPR values for all threshold values corresponding to an AUC value of 0.5.

AUC Diagram

Averaging Type

When you compute a metric, you can aggregate the results into a single value in several ways.

• Micro average — Combine contributions of all observations from each class. For example, calculate the true positive (TP), false positive (FP), true negative (TN), and false negative (FN) across all K classes and then compute the metric.

• Macro average — Calculate the metric for each class and then take the average across all the classes. For example, use the true positive rate for just those observations in class i (TP_i).

• Weighted average — Calculate the metric for each class and then take the weighted average.

Version History

Introduced in R2023b