updateMetricsAndFit

R2026b

Update performance metrics in neural network incremental learning model given new data and train model

Since R2026b

Syntax

Mdl = updateMetricsAndFit(Mdl,X,Y)

Mdl = updateMetricsAndFit(Mdl,X,Y,Name=Value)

[Mdl,ConvergenceInfo] = updateMetricsAndFit(___)

Description

Given streaming data, updateMetricsAndFit first evaluates the performance of a configured neural network incremental learning model for regression (incrementalRegressionNeuralNetwork model object) or classification (incrementalClassificationNeuralNetwork model object) by calling updateMetrics on incoming data. Then updateMetricsAndFit fits the model to that data by calling fit. In other words, updateMetricsAndFit performs prequential evaluation because it treats each incoming chunk of data as a test set, and tracks performance metrics measured cumulatively and over a specified window [1].

updateMetricsAndFit provides a simple way to update model performance metrics and train the model on each chunk of data. Alternatively, you can perform the operations separately by calling updateMetrics and then fit, which allows for more flexibility (for example, you can decide whether you need to train the model based on its performance on a chunk of data).

Mdl = updateMetricsAndFit(Mdl,X,Y) returns an incremental learning model Mdl, which is the input learning model Mdl with the following modifications:

1. updateMetricsAndFit measures the model performance on the incoming predictor and response data, X and Y respectively. When the input model is warm (Mdl.IsWarm is true), updateMetricsAndFit overwrites previously computed metrics, stored in the Metrics property, with the new values. Otherwise, updateMetricsAndFit stores NaN values in Metrics instead.

2. updateMetricsAndFit fits the modified model to the incoming data by updating the neural network weights and biases using the specified solver algorithm, and stores the new parameters in the output model Mdl.

Mdl = updateMetricsAndFit(Mdl,X,Y,Name=Value) uses additional options specified by one or more name-value arguments. For example, you can specify that the columns of the predictor data matrix correspond to observations, and set observation weights.

[Mdl,ConvergenceInfo] = updateMetricsAndFit(___) also returns solver convergence information in the structure ConvergenceInfo, using any of the input arguments from the previous syntaxes.

Examples

Configure Incremental Learning Options

Load the human activity data set. Randomly shuffle the data.

load humanactivity
n = numel(actid);
rng(0,"twister"); % For reproducibility
idx = randsample(n,n);
X = feat(idx,:);
Y = actid(idx);

The class names map 1 through 5 to an activity—sitting, standing, walking, running, or dancing, respectively—based on biometric data measured on the subject. For details on the data set, enter Description at the command line.

Create an incremental neural network model for multiclass learning. Configure the model as follows:

• Specify a metrics warm-up period of 5000 observations.

• Specify a metrics window size of 500 observations.

• Standardize the predictor data and specify an estimation period of 1000 observations.

• Use the mini-batch LBFGS solver and a solver tuning period of 500 observations.

• Double the penalty to the classifier when it mistakenly classifies class 2.

• Track the classification error and minimal cost to measure the performance of the model. You do not have to specify mincost for Metrics because incrementalClassificationNeuralNetwork always tracks this metric.

C = ones(5) - eye(5);
C(2,[1 3 4 5]) = 2;
Mdl = incrementalClassificationNeuralNetwork(ClassNames=1:5, ...
    MetricsWarmupPeriod=5000,MetricsWindowSize=500, ...
    Standardize=true,EstimationPeriod=1000, ...
    TrainingOptions=incrementalTrainingOptions("minibatch-lbfgs", ...
    TuningPeriod=500),Cost=C,Metrics="classiferror")

Mdl = 
  incrementalClassificationNeuralNetwork

                   IsWarm: 0
                  Metrics: [2×2 table]
               ClassNames: [1 2 3 4 5]
           ScoreTransform: 'none'
               LayerSizes: 10
              Activations: "relu"
    OutputLayerActivation: "softmax"
                   Solver: "minibatch-lbfgs"


  Properties, Methods

Mdl is an incrementalClassificationNeuralNetwork model object configured for incremental learning.

Fit the incremental model to the rest of the data by using the updateMetricsAndFit function. At each iteration:

• Simulate a data stream by processing a chunk of 50 observations.

• Overwrite the previous incremental model with a new one fitted to the incoming observations.

• Store the standard deviation of the first predictor variable ${\sigma }_1$, the cumulative metrics, and the window metrics to see how they evolve during incremental learning.

% Preallocation
numObsPerChunk = 50;
nchunk = floor(n/numObsPerChunk);
ce = array2table(zeros(nchunk,2),VariableNames=["Cumulative" "Window"]);
mc = array2table(zeros(nchunk,2),VariableNames=["Cumulative" "Window"]);
sigma1 = zeros(nchunk+1,1);    

% Incremental fitting
for j = 1:nchunk
    ibegin = min(n,numObsPerChunk*(j-1) + 1);
    iend   = min(n,numObsPerChunk*j);
    idx = ibegin:iend;    
    Mdl = updateMetricsAndFit(Mdl,X(idx,:),Y(idx));
    ce{j,:} = Mdl.Metrics{"ClassificationError",:};
    mc{j,:} = Mdl.Metrics{"MinimalCost",:};
    sigma1(j) = Mdl.Sigma(1);
end

Mdl is an incrementalClassificationNeuralNetwork model object trained on all the data in the stream. During incremental learning and after the model is warmed up, updateMetricsAndFit checks the performance of the model on the incoming observations, and then fits the model to those observations.

To see how the performance metrics and ${\sigma }_1$ evolve during training, plot them on separate tiles.

tiledlayout(2,2)
nexttile
plot(sigma1)
ylabel("\sigma_{1}")
xlim([0 nchunk]);
xline(Mdl.EstimationPeriod/numObsPerChunk,"b--")
xlabel("Iteration")
nexttile
h = plot(ce.Variables);
xlim([0 nchunk])
ylabel("Classification Error")
xline((Mdl.EstimationPeriod + Mdl.TrainingOptions.TuningPeriod + ...
 Mdl.MetricsWarmupPeriod)/numObsPerChunk,"r-.")
legend(h,ce.Properties.VariableNames)
xlabel("Iteration")
nexttile
h = plot(mc.Variables);
xlim([0 nchunk]);
ylabel("Minimal Cost")
xline((Mdl.EstimationPeriod + Mdl.TrainingOptions.TuningPeriod + ...
    Mdl.MetricsWarmupPeriod)/numObsPerChunk,"r-.")
legend(h,mc.Properties.VariableNames)
xlabel("Iteration")

The plots indicate that updateMetricsAndFit performs the following actions:

• Fit ${\sigma }_1$ after the estimation period (blue vertical line).

• Compute the performance metrics after the estimation period, tuning period, and metrics warm-up period (red vertical line) only.

• Compute the cumulative metrics during each iteration.

• Compute the window metrics after processing 500 observations (10 iterations).

Specify Orientation of Observations and Observation Weights

Train a neural network classification model by using fitcnet and convert it to an incremental learner by using incrementalLearner. Track the model performance on streaming data and fit the model to streaming data in one call by using updateMetricsAndFit. Specify the orientation of observations and the observation weights when you call updateMetricsAndFit.

Load and Preprocess Data

Load the human activity data set. Randomly shuffle the data.

load humanactivity
rng(0,"twister") % For reproducibility
n = numel(actid);
idx = randsample(n,n);
X = feat(idx,:);
Y = actid(idx);

For details on the data set, enter Description at the command line.

Suppose that the data from a stationary subject (Y <= 2) has double the quality of data from a moving subject. Create a weight variable that assigns a weight of 2 to observations from a stationary subject and 1 to a moving subject.

W = ones(n,1) + (Y <= 2);

Train Neural Network Classification Model

Fit a neural network classification model to a random sample of half the data.

idxtt = randsample([true false],n,true);
TTMdl = fitcnet(X(idxtt,:),Y(idxtt),Weights=W(idxtt))

TTMdl = 
  ClassificationNeuralNetwork
             ResponseName: 'Y'
    CategoricalPredictors: []
               ClassNames: [1 2 3 4 5]
           ScoreTransform: 'none'
          NumObservations: 12039
               LayerSizes: 10
              Activations: 'relu'
    OutputLayerActivation: 'softmax'
                   Solver: 'LBFGS'
          ConvergenceInfo: [1×1 struct]
          TrainingHistory: [1000×7 table]


  Properties, Methods

TTMdl is a ClassificationNeuralNetwork model object representing a traditionally trained neural network classification model.

Convert Trained Model

Convert the traditionally trained model to a model for incremental learning and specify to track the classification error metric.

IncrementalMdl = incrementalLearner(TTMdl,Metrics="classiferror")

IncrementalMdl = 
  incrementalClassificationNeuralNetwork

                   IsWarm: 0
                  Metrics: [2×2 table]
               ClassNames: [1 2 3 4 5]
           ScoreTransform: 'none'
               LayerSizes: 10
              Activations: "relu"
    OutputLayerActivation: "softmax"
                   Solver: "minibatch-lbfgs"


  Properties, Methods

IncrementalMdl is an incrementalClassificationNeuralNetwork model object. Because class names are specified in IncrementalMdl.ClassNames, labels encountered during incremental learning must be in IncrementalMdl.ClassNames.

Track Performance Metrics and Fit Model

Perform incremental learning on the rest of the data by using the updateMetricsAndFit function. Transpose the predictor matrix, and specify the data orientation when you call updateMetricsAndFit. At each iteration:

1. Simulate a data stream by processing 50 observations at a time.

2. Call updateMetricsAndFit to update the cumulative and window performance metrics of the model given the incoming chunk of observations, and then fit the model to the data. Overwrite the previous incremental model with a new one. Specify that the observations are oriented in columns, and specify the observation weights.

3. Store the misclassification error rate.

% Preallocation
idxil = ~idxtt;
nil = sum(idxil);
numObsPerChunk = 50;
nchunk = floor(nil/numObsPerChunk);
mc = array2table(zeros(nchunk,2),VariableNames=["Cumulative","Window"]);
Xil = X(idxil,:)';
Yil = Y(idxil);
Wil = W(idxil);

% Incremental fitting
for j = 1:nchunk
    ibegin = min(nil,numObsPerChunk*(j-1) + 1);
    iend   = min(nil,numObsPerChunk*j);
    idx = ibegin:iend;
    IncrementalMdl = updateMetricsAndFit(IncrementalMdl,Xil(:,idx),Yil(idx), ...
        Weights=Wil(idx),ObservationsIn="columns");
    mc{j,:} = IncrementalMdl.Metrics{"ClassificationError",:};
end

IncrementalMdl is an incrementalClassificationECOC model object trained on all the data in the stream.

Create a trace plot of the misclassification error rate.

plot(mc.Variables)
xlim([0 nchunk])
ylabel("Classification Error")
legend(mc.Properties.VariableNames)
xlabel("Iteration")

The cumulative loss initially has a high value, but stabilizes around 0.05, whereas the window loss jumps throughout the training.

Configure Incremental Learning Options

Prepare an incremental regression learner by specifying a metrics warm-up period and a metrics window size. Train the model by using SGD, and adjust the SGD batch size, learning rate, and regularization parameter.

Load the robot arm data set.

load robotarm

For details on the data set, enter Description at the command line.

Create an incremental neural network model for regression. Configure the model as follows:

• Specify a metrics warm-up period of 1000 observations.

• Specify a metrics window size of 500 observations.

• Specify the FreeRex solver and apply parameter updates based on the L2-norm of the gradient.

• Track the mean squared error (MSE) and mean absolute error (MAE) to measure the performance of the model. Create an anonymous function that measures the absolute error of each new observation. Create a structure array containing the name MeanAbsoluteError and its corresponding function.

maefcn = @(z,zfit)abs(z - zfit);
maemetric = struct("MeanAbsoluteError",maefcn);

Mdl = incrementalRegressionNeuralNetwork(MetricsWarmupPeriod=1000,MetricsWindowSize=500, ...
    TrainingOptions=incrementalTrainingOptions("freerex",UpdateMethod="l2-norm"), ...
    Metrics={"mse",maemetric})

Mdl = 
  incrementalRegressionNeuralNetwork

                   IsWarm: 0
                  Metrics: [2×2 table]
        ResponseTransform: 'none'
               LayerSizes: 10
              Activations: "relu"
    OutputLayerActivation: "none"
                   Solver: "freerex"


  Properties, Methods

Mdl is an incrementalRegressionNeuralNetwork model object configured for incremental learning without an estimation period or solver tuning period.

Fit the incremental model to the data by using the updateMetricsAndFit function. At each iteration:

• Simulate a data stream by processing a chunk of 50 observations.

• Overwrite the previous incremental model with a new one fitted to the incoming observations.

• Store the cumulative metrics, window metrics, and number of training observations to see how they evolve during incremental learning.

% Preallocation
n = numel(ytrain);
numObsPerChunk = 50;
nchunk = floor(n/numObsPerChunk);
mse = array2table(zeros(nchunk,2),VariableNames=["Cumulative","Window"]);
mae = array2table(zeros(nchunk,2),VariableNames=["Cumulative","Window"]);  
numtrainobs = zeros(nchunk,1);

% Incremental fitting
rng(0,"twister") % For reproducibility
for j = 1:nchunk
    ibegin = min(n,numObsPerChunk*(j-1) + 1);
    iend   = min(n,numObsPerChunk*j);
    idx = ibegin:iend;    
    Mdl = updateMetricsAndFit(Mdl,Xtrain(idx,:),ytrain(idx));
    mse{j,:} = Mdl.Metrics{"MeanSquaredError",:};
    mae{j,:} = Mdl.Metrics{"MeanAbsoluteError",:};
    numtrainobs(j) = Mdl.NumTrainingObservations;
end

Mdl is an incrementalRegressionNeuralNetwork model object trained on all the data in the stream. During incremental learning and after the model is warmed up, updateMetricsAndFit checks the performance of the model on the incoming observations, and then fits the model to those observations.

Plot a trace plot of the number of training observations and the performance metrics on separate tiles.

t = tiledlayout(3,1);
nexttile
plot(numtrainobs)
xlim([0 nchunk])
ylabel(["Number of","Training Observations"])
xline(Mdl.MetricsWarmupPeriod/numObsPerChunk,"--")
nexttile
plot(mse.Variables)
xlim([0 nchunk])
ylabel("MSE")
xline(Mdl.MetricsWarmupPeriod/numObsPerChunk,"--")
legend(mse.Properties.VariableNames)
nexttile
plot(mae.Variables)
xlim([0 nchunk])
ylabel("MAE")
xline(Mdl.MetricsWarmupPeriod/numObsPerChunk,"--")
legend(mae.Properties.VariableNames)
xlabel(t,"Iteration")

The plot suggests that updateMetricsAndFit does the following:

• Fit the model during all incremental learning iterations.

• Compute the performance metrics after the metrics warm-up period only (dashed vertical line).

• Compute the cumulative metrics during each iteration.

• Compute the window metrics after processing 500 observations (10 iterations).

Input Arguments

Mdl — Incremental learning model

incrementalClassificationNeuralNetwork model object | incrementalRegressionNeuralNetwork model object

Incremental learning model whose performance is measured and then the model is fit to data, specified as an incrementalClassificationNeuralNetwork or incrementalRegressionNeuralNetwork model object. You can create Mdl directly or by converting a supported, traditionally trained machine learning model using the incrementalLearner function. For more details, see the corresponding reference page.

If Mdl.IsWarm is false, updateMetricsAndFit does not track the performance of the model. For more details, see Performance Metrics.

X — Chunk of predictor data

floating-point matrix

Chunk of predictor data, specified as a floating-point matrix of n observations and Mdl.NumPredictors predictor variables. The value of the ObservationsIn name-value argument determines the orientation of the variables and observations. The default ObservationsIn value is "rows", which indicates that observations in the predictor data are oriented along the rows of X.

The length of the observation responses (labels) Y and the number of observations in X must be equal; Y(j) is the response (label) of observation j (row or column) in X.

Note

• updateMetricsAndFit supports only floating-point input predictor data. If your input data includes categorical data, you must prepare an encoded version of the categorical data. Use dummyvar to convert each categorical variable to a numeric matrix of dummy variables. Then, concatenate all dummy variable matrices and any other numeric predictors. For more details, see Dummy Variables.

Data Types: single | double

Y — Chunk of responses (labels)

categorical array | character array | string array | logical vector | floating-point vector | cell array of character vectors

Chunk of responses (labels), specified as a categorical, character, or string array, a logical or floating-point vector, or a cell array of character vectors for classification problems; or a floating-point vector for regression problems.

The length of the observation responses Y and the number of observations in X must be equal; Y(j) is the response of observation j (row or column) in X.

For classification problems, updateMetricsAndFit issues an error when one or both of these conditions are met:

• Y contains a new label and the maximum number of classes has already been reached (see the ClassNames and MaxNumClasses arguments of incrementalClassificationNeuralNetwork).

• The ClassNames property of the input model Mdl is nonempty, and the data types of Y and Mdl.ClassNames are different.

Data Types: char | string | cell | categorical | logical | single | double

Note

If an observation (predictor or label) or weight contains at least one missing (NaN) value, updateMetricsAndFit ignores the observation. Consequently, updateMetricsAndFit uses fewer than n observations to compute the model performance and create an updated model, where n is the number of observations in X.

Name-Value Arguments

Specify optional pairs of arguments as Name1=Value1,...,NameN=ValueN, where Name is the argument name and Value is the corresponding value. Name-value arguments must appear after other arguments, but the order of the pairs does not matter.

Example: updateMetricsAndFit(Mdl,X,Y,ObservationsIn="columns",Weights=W) specifies that the columns of the predictor matrix correspond to observations, and the vector W contains observation weights to apply during incremental learning.

ObservationsIn — Predictor data observation dimension

"rows" (default) | "columns"

Predictor data observation dimension, specified as "rows" or "columns".

Example: ObservationsIn="columns"

Data Types: char | string

Weights — Chunk of observation weights

floating-point vector of positive values

Chunk of observation weights, specified as a floating-point vector of positive values. updateMetricsAndFit weighs the observations in X with the corresponding values in Weights. The size of Weights must equal n, which is the number of observations in X.

By default, Weights is ones(n,1).

For more details, including normalization schemes, see Observation Weights.

Example: Weights=W specifies the observation weights as the vector W.

Data Types: double | single

Output Arguments

Mdl — Updated incremental learning model

incrementalClassificationNeuralNetwork model object | incrementalRegressionNeuralNetwork model object

Updated incremental learning model, returned as an incremental learning model object of the same data type as the input model Mdl, either incrementalClassificationNeuralNetwork or incrementalRegressionNeuralNetwork.

If the model is not warm, updateMetricsAndFit does not compute performance metrics. As a result, the Metrics property of Mdl remains completely composed of NaN values. If the model is warm, updateMetricsAndFit computes the cumulative and window performance metrics on the new data X and Y, and overwrites the corresponding elements of Mdl.Metrics. For more details, see Performance Metrics.

After updating metrics, updateMetricsAndFit trains the model on the incoming data. Specifically, it updates the LayerWeights, LayerBiases, and NumTrainingObservations properties.

ConvergenceInfo — Solver convergence information

structure

Solver convergence information, returned as a structure containing the following fields:

• GradientsNorm — A real nonnegative scalar specifying the L-infinity norm of the gradient at the last iteration.

• StepNorm — A real nonnegative scalar specifying the L2 norm of the step taken at the last iteration.

• Gradients — A cell array of numeric matrices specifying the gradients of the loss with respect to the parameters at the last iteration. Gradients has 2*K cells, where K is the number of cells in the LayerWeights property of Mdl. The first K cells correspond to the gradients with respect to Mdl.LayerWeights, and the remaining cells correspond to the gradients with respect to Mdl.LayerBiases.

During the estimation and solver tuning periods, the function returns NaN values for all fields.

Algorithms

Solver Algorithms

The updateMetricsAndFit function uses the solver specified in Mdl.TrainingOptions to update the neural network weights and biases. The supported solvers are:

• "minibatch-lbfgs" — A mini-batch L-BFGS (limited-memory Broyden-Fletcher-Goldfarb-Shanno) solver that processes the incoming data chunk as a mini-batch and performs multiple iterations of L-BFGS optimization on it.

• "freerex" — An adaptive, scale-invariant online learning algorithm that does not require tuning a learning rate.

You can configure training options, including the solver choice, by using the incrementalTrainingOptions function and passing the result to the TrainingOptions name-value argument when creating the model.

Performance Metrics

• The updateMetrics and updateMetricsAndFit functions track model performance metrics from new data only when the incremental model is warm (IsWarm property is true).

  The Metrics property of the incremental model stores two forms of each performance metric as variables (columns) of a table, Cumulative and Window, with individual metrics in rows. When the incremental model is warm, updateMetrics and updateMetricsAndFit update the metrics at the following frequencies:

  • Cumulative — The functions compute cumulative metrics since the start of model performance tracking. The functions update metrics every time you call the functions and base the calculation on the entire supplied data set.

  • Window — The functions compute metrics based on all observations within a window determined by the MetricsWindowSize name-value argument. MetricsWindowSize also determines the frequency at which the software updates Window metrics. For example, if MetricsWindowSize is 20, the functions compute metrics based on the last 20 observations in the supplied data (X((end – 20 + 1):end,:) and Y((end – 20 + 1):end)).

    Incremental functions that track performance metrics within a window use the following process:

    1. Store a buffer of length MetricsWindowSize for each specified metric, and store a buffer of observation weights.

    2. Populate elements of the metrics buffer with the model performance based on batches of incoming observations, and store corresponding observation weights in the weights buffer.

    3. When the buffer is full, overwrite Mdl.Metrics.Window with the weighted average performance in the metrics window. If the buffer overfills when the function processes a batch of observations, the latest incoming MetricsWindowSize observations enter the buffer, and the earliest observations are removed from the buffer. For example, suppose MetricsWindowSize is 20, the metrics buffer has 10 values from a previously processed batch, and 15 values are incoming. To compose the length 20 window, the functions use the measurements from the 15 incoming observations and the latest 5 measurements from the previous batch.

The incremental fitting functions omit an observation with a NaN score when computing the Cumulative and Window performance metric values.

Incremental Training Periods

When you train an incremental neural network model with the incremental fitting functions fit and updateMetricsAndFit, then depending on the model's properties, up to three incremental training periods can occur in the following order: the estimation period, the solver tuning period, and the metrics warm-up period. Following these periods, the incremental model is warm and the incremental fitting functions track model performance metrics from new data.

Estimation Period

During the estimation period, fit does not fit the model, and updateMetricsAndFit does not fit the model or update the performance metrics. The incremental fitting functions use the first incoming EstimationPeriod observations to estimate the predictor means and standard deviation hyperparameters required to standardize the data during incremental training. The fitting functions store the hyperparameter estimates in the Mu and Sigma properties of Mdl.

The hyperparameters are estimated when both of these conditions apply:

• Incremental fitting functions are configured to standardize predictor data (see Standardize Data).

• Mu and Sigma are empty arrays [].

When you create the model object using the incrementalLearner function, EstimationPeriod is always 0.

Solver Tuning Period

During the solver tuning period, the incremental fitting functions use Mdl.TrainingOptions.TuningPeriod observations to tune the parameters of the mini-batch LBFGS solver (the default solver). There is no solver turning period for the FreeREX solver. You can select the solver algorithm and the length of the solver tuning period using the TrainingOptions name-value argument when you create the model object. For more information, see the Limited-Memory BFGS and FreeRex sections of the incrementalTrainingOptions reference page.

Metrics Warm-Up Period

During the metrics warm-up period, the incremental fitting functions fit the incremental model.

• An incrementalClassificationNeuralNetwork model object is warm and tracks the performance metrics in its Metrics property after the incremental fitting functions process MetricsWarmupPeriod observations and fit at least one observation from each expected class (see the MaxNumClasses and ClassNames arguments of incrementalClassificationNeuralNetwork).

• An incrementalRegressionNeuralNetwork model object is warm after the incremental fitting functions process MetricsWarmupPeriod observations.

Observation Weights

For classification problems, if the prior class probability distribution is known (in other words, the prior distribution is not empirical), updateMetricsAndFit normalizes observation weights to sum to the prior class probabilities in the respective classes for updating the metrics. This action implies that the default observation weights are the respective prior class probabilities.

For regression problems or if the prior class probability distribution is empirical, the following applies: The software normalizes the specified observation weights to sum to 1 each time you call updateMetricsAndFit to update the metrics.

References

[1] Bifet, Albert, Ricard Gavaldá, Geoffrey Holmes, and Bernhard Pfahringer. Machine Learning for Data Streams with Practical Example in MOA. Cambridge, MA: The MIT Press, 2007.

Version History

Introduced in R2026b