modelAccuracy
Compute R-square, RMSE, correlation, and sample mean error of predicted and observed EADs
Description
computes the R-square, root mean square error (RMSE), correlation, and sample mean
error of observed vs. predicted exposure at default (EAD) data.
AccMeasure
= modelAccuracy(eadModel
,data
)modelAccuracy
supports comparison against a reference model
and also supports different correlation types. By default,
modelAccuracy
computes the metrics in the EAD scale. You can
use the ModelLevel
name-value argument to compute metrics using
the underlying model's transformed scale.
[
specifies options using one or more name-value arguments in addition to the input
arguments in the previous syntax.AccMeasure
,AccData
] = modelAccuracy(___,Name=Value
)
Examples
Compute R-Square, RMSE, Correlation, and Sample Mean Error of Predicted and Observed Using a Tobit EAD Model
This example shows how to use fitEADModel
to create a Tobit
model and then use modelAccuracy
to compute the R-Square, RMSE, correlation, and sample mean error of predicted and observed EAD.
Load EAD Data
Load the EAD data.
load EADData.mat
head(EADData)
ans=8×6 table
UtilizationRate Age Marriage Limit Drawn EAD
_______________ ___ ___________ __________ __________ __________
0.24359 25 not married 44776 10907 44740
0.96946 44 not married 2.1405e+05 2.0751e+05 40678
0 40 married 1.6581e+05 0 1.6567e+05
0.53242 38 not married 1.7375e+05 92506 1593.5
0.2583 30 not married 26258 6782.5 54.175
0.17039 54 married 1.7357e+05 29575 576.69
0.18586 27 not married 19590 3641 998.49
0.85372 42 not married 2.0712e+05 1.7682e+05 1.6454e+05
rng('default'); NumObs = height(EADData); c = cvpartition(NumObs,'HoldOut',0.4); TrainingInd = training(c); TestInd = test(c);
Select Model Type
Select a model type for Tobit
or Regression
.
ModelType =
"Tobit";
Select Conversion Measure
Select a conversion measure for the EAD response values.
ConversionMeasure =
"LCF";
Create Tobit
EAD Model
Use fitEADModel
to create a Tobit
model using EADData
.
eadModel = fitEADModel(EADData,ModelType,PredictorVars={'UtilizationRate','Age','Marriage'}, ... ConversionMeasure=ConversionMeasure,DrawnVar="Drawn",LimitVar="Limit",ResponseVar="EAD"); disp(eadModel);
Tobit with properties: CensoringSide: "both" LeftLimit: 0 RightLimit: 1 ModelID: "Tobit" Description: "" UnderlyingModel: [1x1 risk.internal.credit.TobitModel] PredictorVars: ["UtilizationRate" "Age" "Marriage"] ResponseVar: "EAD" LimitVar: "Limit" DrawnVar: "Drawn" ConversionMeasure: "lcf"
Display the underlying model. The underlying model's response variable is the transformation of the EAD response data. Use the 'LimitVar'
and 'DrwanVar'
name-value arguments to modify the transformation.
disp(eadModel.UnderlyingModel);
Tobit regression model: EAD_lcf = max(0,min(Y*,1)) Y* ~ 1 + UtilizationRate + Age + Marriage Estimated coefficients: Estimate SE tStat pValue __________ __________ ________ ________ (Intercept) 0.22735 0.025005 9.0922 0 UtilizationRate 0.47364 0.016531 28.652 0 Age -0.0013929 0.00061479 -2.2657 0.023517 Marriage_not married -0.006888 0.01208 -0.57022 0.56856 (Sigma) 0.36419 0.003878 93.913 0 Number of observations: 4378 Number of left-censored observations: 0 Number of uncensored observations: 4377 Number of right-censored observations: 1 Log-likelihood: -1791.06
Predict EAD
EAD prediction operates on the underlying compact statistical model and then transforms the predicted values back to the EAD scale. You can specify the predict
function with different options for the 'ModelLevel'
name-value argument.
predictedEAD = predict(eadModel,EADData(TestInd,:),ModelLevel="ead"); predictedConversion = predict(eadModel,EADData(TestInd,:),ModelLevel="ConversionMeasure");
Validate EAD Model
For model validation, use modelDiscrimination
, modelDiscriminationPlot
, modelAccuracy
, and modelAccuracyPlot
.
Use modelDiscrimination
and then modelDiscriminationPlot
to plot the ROC curve.
ModelLevel ="ead"; [DiscMeasure1,DiscData1] = modelDiscrimination(eadModel,EADData(TestInd,:),ModelLevel=ModelLevel); modelDiscriminationPlot(eadModel,EADData(TestInd, :),ModelLevel=ModelLevel,SegmentBy="Marriage");
Use modelAccuracy
and then modelAccuracyPlot
to show a scatter plot of the predictions.
YData =
"Observed";
[AccMeasure1,AccData1] = modelAccuracy(eadModel,EADData(TestInd,:),ModelLevel=ModelLevel)
AccMeasure1=1×4 table
RSquared RMSE Correlation SampleMeanError
________ _____ ___________ _______________
Tobit 0.39127 42545 0.62552 -1713.1
AccData1=1751×3 table
Observed Predicted_Tobit Residuals_Tobit
__________ _______________ _______________
44740 15177 29563
54.175 8900.3 -8846.1
987.39 13430 -12443
9606.4 7422.4 2184
83.809 27852 -27768
73538 46229 27309
96.949 5582.8 -5485.9
873.21 4527.1 -3653.9
328.35 6079.8 -5751.5
55237 28295 26942
30359 19177 11182
39211 28753 10457
2.0885e+05 1.0725e+05 1.016e+05
1921.7 20132 -18210
15230 5526.4 9703.5
20063 9501.2 10562
⋮
modelAccuracyPlot(eadModel,EADData(TestInd,:),ModelLevel=ModelLevel,YData=YData);
Input Arguments
eadModel
— Exposure at default model
Regression
or Tobit
object
Loss given default model, specified as a previously created Regression
or
Tobit
object using
fitEADModel
.
Data Types: object
data
— Data
table
Data, specified as a
NumRows
-by-NumCols
table with
predictor and response values. The variable names and data types must be
consistent with the underlying model.
Data Types: table
Name-Value Arguments
Specify optional pairs of arguments as
Name1=Value1,...,NameN=ValueN
, where Name
is
the argument name and Value
is the corresponding value.
Name-value arguments must appear after other arguments, but the order of the
pairs does not matter.
Example: [AccMeasure,AccData] =
modelAccuracy(eadModel,data(TestInd,:),DataID='Testing',CorrelationType='spearman')
CorrelationType
— Correlation type
"pearson"
(default) | character vector with value of 'pearson'
,
'spearman'
, or
'kendall'
| string with value of "pearson"
,
"spearman"
, or
"kendall"
Correlation type, specified as CorrelationType
and
a character vector or string.
Data Types: char
| string
DataID
— Data set identifier
""
(default) | character vector | string
Data set identifier, specified as DataID
and a
character vector or string. The DataID
is included in
the output for reporting purposes.
Data Types: char
| string
ModelLevel
— Model level
'ead'
(default) | character vector with value 'ead'
,
'conversionMeasure'
, or
'conversionTransform'
| string with value "ead"
,
"conversionMeasure"
, or
"conversionTransform"
Model level, specified as ModelLevel
and a
character vector or string.
Note
Regression
models support all three model levels,
but a Tobit
model supports model levels only for "ead"
and "conversionMeasure"
.
Data Types: char
| string
ReferenceEAD
— EAD values predicted for data
by reference model
[]
(default) | numeric vector
ReferenceID
— Identifier for the reference model
'Reference'
(default) | character vector | string
Identifier for the reference model, specified as
ReferenceID
and a character vector or string.
ReferenceID
is used in the
modelAccuracy
output for reporting
purposes.
Data Types: char
| string
Output Arguments
AccMeasure
— Accuracy measure
table
Accuracy measure, returned as a table with columns
'RSquared'
, 'RMSE'
,
'Correlation'
, and
'SampleMeanError'
. AccMeasure
has
one row if only the eadModel
accuracy is measured and
it has two rows if reference model information is given. The row names of
AccMeasure
report the model ID and data ID (if
provided).
AccData
— Accuracy data
table
Accuracy data, returned as a table with observed EAD values, predicted EAD
values, and residuals (observed minus predicted). Additional columns for
predicted and residual values are included for the reference model, if
provided. The ModelID
and
ReferenceID
labels are appended in the column
names.
More About
Model Accuracy
Model accuracy measures the accuracy of the predicted probability of EAD values using different metrics.
R-squared — To compute the R-squared metric,
modelAccuracy
fits a linear regression of the observed EAD values against the predicted EAD values:The R-square of this regression is reported. For more information, see Coefficient of Determination (R-Squared).
RMSE — To compute the root mean square error (RMSE),
modelAccuracy
uses the following formula where N is the number of observations:Correlation — This metric is the correlation between the observed and predicted EAD:
For more information and details about the different correlation types, see
corr
.Sample mean error — This metric is the difference between the mean observed EAD and the mean predicted EAD or, equivalently, the mean of the residuals:
References
[1] Baesens, Bart, Daniel Roesch, and Harald Scheule. Credit Risk Analytics: Measurement Techniques, Applications, and Examples in SAS. Wiley, 2016.
[2] Bellini, Tiziano. IFRS 9 and CECL Credit Risk Modelling and Validation: A Practical Guide with Examples Worked in R and SAS. San Diego, CA: Elsevier, 2019.
[3] Brown, Iain. Developing Credit Risk Models Using SAS Enterprise Miner and SAS/STAT: Theory and Applications. SAS Institute, 2014.
[4] Roesch, Daniel and Harald Scheule. Deep Credit Risk. Independently published, 2020.
Version History
Apri esempio
Si dispone di una versione modificata di questo esempio. Desideri aprire questo esempio con le tue modifiche?
Comando MATLAB
Hai fatto clic su un collegamento che corrisponde a questo comando MATLAB:
Esegui il comando inserendolo nella finestra di comando MATLAB. I browser web non supportano i comandi MATLAB.
Select a Web Site
Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .
You can also select a web site from the following list:
How to Get Best Site Performance
Select the China site (in Chinese or English) for best site performance. Other MathWorks country sites are not optimized for visits from your location.
Americas
- América Latina (Español)
- Canada (English)
- United States (English)
Europe
- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)
- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- United Kingdom (English)