loss

Evaluate accuracy of learned feature weights on test data

Syntax

err = loss(mdl,Tbl,ResponseVarName)

err = loss(mdl,Tbl,Y)

err = loss(mdl,X,Y)

err = loss(___,LossFunction=lossFunc)

Description

err = loss(mdl,Tbl,ResponseVarName) returns the mean squared error as the measure of accuracy in err, for the model mdl, predictor values in table Tbl, and the variable ResponseVarName in table Tbl. Use this syntax if mdl was originally trained on a table.

err = loss(mdl,Tbl,Y) returns the mean squared error as the measure of accuracy in err, for the model mdl, predictor values in table Tbl, and response values in Y. Use this syntax if mdl was originally trained on a table.

err = loss(mdl,X,Y) returns the mean squared error as the measure of accuracy in err, for the model mdl, predictor values in matrix X, and response values in Y. Use this syntax if mdl was originally trained on a numeric matrix.

err = loss(___,LossFunction=lossFunc) returns the measure of accuracy according to the specified loss function type. Use the loss function specification in addition to the input arguments in previous syntaxes.

Examples

Select Features for Neighborhood Component Analysis (NCA) Model

Load the robotarm data set.

load robotarm.mat

The robotarm data set was created using a robot arm simulator with 7168 training observations and 1024 test observations with 32 features [1][2]. The data set used in this example is a preprocessed version of the original data set. The data was preprocessed by subtracting values given by a linear regression fit from the original data, followed by normalization of all features to unit variance.

Perform NCA feature selection for regression with the default regularization parameter value λ.

rng(1,"twister");   % For reproducibility
nca = fsrnca(Xtrain,ytrain);

Calculate the loss for the model using the test set as a measure of the performance.

l = loss(nca,Xtest,ytest)

l = 
0.0844

By default, the function calculates the mean squared error.

Select features using feature weights and a relative threshold.

fw = nca.FeatureWeights;
tol = 0.001;
threshold = max(fw)*tol;
vars = fw>threshold;

Create new training and test sets for the predictor data using only the selected features.

Xtrain2 = Xtrain(:,vars);
Xtest2 = Xtest(:,vars);

Update the NCA model using the new training set. Calculate the loss for the new model using the new test set.

nca2 = fsrnca(Xtrain2,ytrain);
l2 = loss(nca2,Xtest2,ytest)

l2 = 
0.0845

Calculate the difference in the loss for the original model and the loss for the updated model.

abs(l-l2)

ans = 
1.0549e-04

The small difference in the loss values indicates that the updated model trained with a subset of the original model features, performs similarly to the original model.

Input Arguments

mdl — Neighborhood component analysis model for regression

FeatureSelectionNCARegression object

Neighborhood component analysis model for regression, specified as a FeatureSelectionNCARegression object.

Tbl — Sample data

table

Sample data, specified as a table. Each row of Tbl corresponds to one observation, and each column corresponds to one predictor variable.

Data Types: table

ResponseVarName — Response variable name

name of variable in Tbl

Response variable name, specified as the name of a variable in Tbl. The remaining variables in the table are predictors.

Data Types: char | string

X — Predictor variable values

n-by-p matrix

Predictor variable values, specified as an n-by-p matrix, where n is the number of observations and p is the number of predictor variables used for training the model mdl.

Data Types: single | double

Y — Response values

numeric real vector of length n

Response values, specified as a numeric real vector of length n, where n is the number of observations. You can omit Y if it is included in the table Tbl.

Data Types: single | double

lossFunc — Loss function type

"mse" (default) | "mad"

Loss function type, specified as one of these values:

Loss Function Type	Description
"mse"	Mean squared error
"mad"	Mean absolute deviation

Example: "mse"

Output Arguments

err — Smaller-the-better accuracy measure for learned feature weights

scalar value

Smaller-the-better accuracy measure for learned feature weights, returned as a scalar value. You can specify the measure of accuracy using the LossFunction name-value argument.

References

[1] Rasmussen, C. E., R. M. Neal, G. E. Hinton, D. van Camp, M. Revow, Z. Ghahramani, R. Kustra, and R. Tibshirani. The DELVE Manual, 1996, https://mlg.eng.cam.ac.uk/pub/pdf/RasNeaHinetal96.pdf.

[2] University of Toronto, Computer Science Department. Delve Datasets. http://www.cs.toronto.edu/~delve/data/datasets.html.

Version History

Introduced in R2016b