risk.validation.areaUnderCurveTest

R2026b

Area under curve test

Since R2026a

Syntax

hAUCTest = risk.validation.areaUnderCurveTest(BaselineScore,BaselineBinaryResponse,TargetScore,TargetBinaryResponse)

hAUCTest = risk.validation.areaUnderCurveTest(BaselineScore,BaselineBinaryResponse,TargetScore,TargetBinaryResponse,Name=Value)

[hAUCTest,AUCOutput] = risk.validation.areaUnderCurveTest(___)

Description

hAUCTest = risk.validation.areaUnderCurveTest(BaselineScore,BaselineBinaryResponse,TargetScore,TargetBinaryResponse) returns the result of an area under curve (AUC) test, hAUCTest, which compares scores and responses in a baseline and target portfolio. The output is 1 if the test rejects the null hypothesis at the 95% confidence level, or 0 otherwise.

hAUCTest = risk.validation.areaUnderCurveTest(BaselineScore,BaselineBinaryResponse,TargetScore,TargetBinaryResponse,Name=Value) specifies additional options using one or more name-value arguments. For example, you can specify the confidence level for the test.

[hAUCTest,AUCOutput] = risk.validation.areaUnderCurveTest(___) also returns a structure AUCOutput that contains a table of summary metrics. Specify AUCOutput as the second output argument with any of the input argument combinations in the previous syntaxes.

Examples

Apply AUC Test on Probability of Default Data

Perform an AUC test on customer probability of default (PD) data to determine if the AUC for a baseline portfolio is smaller than the AUC for a target portfolio. The PDRatingGradeData file contains a table with variables PD, RatingGrade, ObservationYear, and Default. These variables, respectively, contain data for customer probability of default (PD), rating grade, observation year, and whether the customer defaulted.

Load the data.

data = readtable("PDRatingGradeData.csv")

data = 500×4 table
       PD       RatingGrade    ObservationYear    Default
    ________    ___________    _______________    _______

     0.40736         9              2020             0   
      0.4529        10              2020             0   
    0.063493         2              2020             0   
     0.45669        10              2020             1   
     0.31618         7              2020             0   
     0.04877         1              2020             0   
     0.13925         3              2020             0   
     0.27344         6              2020             0   
     0.47875        10              2020             0   
     0.48244        10              2020             0   
    0.078807         2              2020             0   
      0.4853        10              2020             0   
     0.47858        10              2020             0   
     0.24269         5              2020             0   
     0.40014         9              2020             0   
    0.070943         2              2020             0   
      ⋮

Create two logical vectors indicating the observations that belong to the baseline portfolio and the observations that belong to the target portfolio.

baselineElem = data.ObservationYear==2023;
targetElem = data.ObservationYear==2024;

Create vectors from the rating grade and binary response data for the baseline and target portfolios.

BaselineScore = data.RatingGrade(baselineElem);
TargetScore = data.RatingGrade(targetElem);
BaselineBinaryResponse = data.Default(baselineElem);
TargetBinaryResponse = data.Default(targetElem);

Calculate the x- and y-coordinates of the receiver operating characteristic (ROC) curve for each portfolio by using the perfcurve function. Plot the ROC curves together.

[XB,YB] = perfcurve(BaselineBinaryResponse,BaselineScore,1);
[XT,YT] = perfcurve(TargetBinaryResponse,TargetScore,1);
plot(XB,YB,"o-")
hold on
plot(XT,YT,"o-")
xlabel("False Positive Rate (FPR)")
ylabel("True Positive Rate (TPR)")
legend("Baseline (2023) portfolio","Target (2024) portfolio",Location="northwest")

The plot shows that the curve for the target portfolio is mostly below the curve for the baseline portfolio.

Use the risk.validation.areaUnderCurveTest function with the fully qualified namespace risk.validation to perform an area under curve test on the baseline and target portfolio data.

risk.validation.areaUnderCurveTest(BaselineScore,BaselineBinaryResponse,TargetScore,TargetBinaryResponse)

ans = logical
   1

The output shows that enough evidence exists to reject the null hypothesis that the AUC for the baseline portfolio is smaller than the AUC for the target portfolio. This result is consistent with the ROC curve for the target portfolio being under the ROC curve for the baseline portfolio.

Input Arguments

BaselineScore — Baseline portfolio scores

numeric vector

Baseline portfolio scores, specified as a numeric vector. BaselineScore must be of the same data type as TargetScore and the same size as BaselineBinaryResponse.

The baseline scores are sometimes called the initial scores.

BaselineBinaryResponse — Baseline portfolio responses

logical vector | numeric vector containing values of 1 (true) or 0 (false)

Baseline portfolio responses, specified as a logical vector or a numeric vector containing values of 1 (true) or 0 (false). The baseline responses, sometimes called the initial responses, represent the predicted state for each value in BaselineScore. For example, you can use the baseline responses to represent whether a member of the portfolio has defaulted.

BaselineBinaryResponse and BaselineScore must be the same size.

TargetScore — Target portfolio scores

numeric vector

Target portfolio scores, specified as a numeric vector. TargetScore must be of the same type as BaselineScore and the same size as TargetBinaryResponse.

The target scores are sometimes called the current scores.

TargetBinaryResponse — Target portfolio responses

logical vector | numeric vector containing values of 1 (true) or 0 (false)

Target portfolio responses, specified as a logical vector or a numeric vector containing values of 1 (true) or 0 (false). The target responses, sometimes called the current responses, represent the predicted state for each value in TargetScore. For example, you can use the target responses to represent customer defaults.

TargetBinaryResponse and TargetScore must be the same size.

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: risk.validation.areaUnderCurveTest(BaselineScore,BaselineBinaryResponse,TargetScore,TargetBinaryResponse,ConfidenceLevel=0.99,SortDirection="ascending") specifies a 99% confidence level for the test and indicates that lower credit scores indicate greater risk.

ConfidenceLevel — Confidence level

0.95 (default) | numeric scalar in the range (0,1)

Confidence level of the hypothesis test, specified as a numeric scalar in the range (0,1).

SortDirection — Direction of increasing risk

"descending" (default) | "ascending"

Direction of increasing risk, specified as "descending" or "ascending". When you specify SortDirection as "descending", higher scores indicate a greater risk of an event. When you specify SortDirection as "ascending", lower scores indicate greater risk.

For example, if BaselineScore and TargetScore contain credit scores, SortDirection should be "ascending" because lower scores indicate higher risk. On the other hand, if BaselineScore and TargetScore contain probability of default (PD) values, SortDirection should be "descending" because higher probabilities indicate higher risk.

Example: SortDirection="ascending"

Data Types: string | char

Output Arguments

hAUCTest — Hypothesis test result

1 | 0

Hypothesis test result, returned as a 1 or 0 of data type logical.

• A value of 1 rejects the null hypothesis at the specified confidence level.

• A value of 0 fails to reject the null hypothesis at the specified confidence level.

AUCOutput — Output metrics

structure

Output metrics, returned as a structure with the following fields:

• RejectTest — Logical scalar indicating whether the null hypothesis was rejected. This field represents the same values as hAUCTest.

• PValue — Numeric scalar with values in the range [0,1] representing the p-value for the hypothesis test. A small value indicates that the null hypothesis might not be valid.

• TestStatistic — Numeric scalar representing the value of the test statistic for the hypothesis test.

• CriticalValue — Numeric scalar representing the minimum test statistic at which the test rejects the null hypothesis for the given probability and confidence level.

• BaselineAUC — Numeric scalar representing the AUC for the baseline portfolio.

• BaselineAUCVariance — Numeric scalar representing the estimated variance of the AUC for the baseline portfolio.

• BaselineTrials — Numeric scalar representing the number of observations in the baseline portfolio.

• TargetAUC — Numeric scalar representing the AUC for the target portfolio.

• TargetAUCVariance — Numeric scalar representing the estimated variance of the AUC for the target portfolio.

• TargetTrials — Numeric scalar representing the number of observations in the target portfolio.

• MeanBaselineScore — Numeric scalar representing the mean of the observations in BaselineScore.

• MeanBaselineResponse — Numeric scalar representing the mean of the observations in BaselineBinaryResponse.

• MeanTargetScore — Numeric scalar representing the mean of the observations in TargetScore.

• MeanTargetResponse — Numeric scalar representing the mean of the observations in TargetBinaryResponse.

• ConfidenceLevel — Numeric scalar representing the confidence level for the hypothesis test.

For more information about the AUC test and its corresponding statistics, see More About.

More About

Current AUC vs. AUC at Initial Validation Test

The current AUC vs. AUC at initial validation test [1], also referred to as the AUC test, requires you to input scores and responses for a baseline portfolio and a target portfolio. The null hypothesis is that the AUC for the baseline portfolio is smaller than the AUC for the target portfolio.

The test statistic for the AUC test is given by

$$S=\frac{BaselineAUC-TargetAUC}{\sqrt{TargetAUCVariance}}\text{,}$$

where BaselineAUC and TargetAUC are the AUCs for the baseline and target portfolios, respectively, and TargetAUCVariance is the estimated variance for the target portfolio. The software estimates the target AUC variance using the equations described in Section 2.5.4.1 of [1].

The p-value for S is equal to $$1-F\left(S\right)$$, where F is the cumulative distribution function for the standard normal distribution.

References

[1] European Central Bank. “Instructions for reporting the validation results of internal models.” February, 2019. https://www.bankingsupervision.europa.eu/activities/internal_models/shared/pdf/instructions_validation_reporting_credit_risk.en.pdf.

Version History

Introduced in R2026a