fitcensemble
Fit ensemble of learners for classification
Syntax
Description
Mdl = fitcensemble(Tbl,ResponseVarName)Mdl) that contains the results of boosting 100
classification trees and the predictor and response data in the table
Tbl. ResponseVarName is the name of
the response variable in Tbl. By default,
fitcensemble uses LogitBoost for binary classification
and AdaBoostM2 for multiclass classification.
Mdl = fitcensemble(Tbl,formula)formula to fit the model to the predictor and
response data in the table Tbl. formula is
an explanatory model of the response and a subset of predictor variables in
Tbl used to fit Mdl. For example,
'Y~X1+X2+X3' fits the response variable
Tbl.Y as a function of the predictor variables
Tbl.X1, Tbl.X2, and
Tbl.X3.
Mdl = fitcensemble(___,Name,Value)Name,Value
pair arguments and any of the input arguments in the previous syntaxes. For
example, you can specify the number of learning cycles, the ensemble aggregation
method, or to implement 10-fold cross-validation.
Examples
Train Classification Ensemble
Create a predictive classification ensemble using all available predictor variables in the data. Then, train another ensemble using fewer predictors. Compare the in-sample predictive accuracies of the ensembles.
Load the census1994 data set.
load census1994Train an ensemble of classification models using the entire data set and default options.
Mdl1 = fitcensemble(adultdata,'salary')Mdl1 =
ClassificationEnsemble
PredictorNames: {1x14 cell}
ResponseName: 'salary'
CategoricalPredictors: [2 4 6 7 8 9 10 14]
ClassNames: [<=50K >50K]
ScoreTransform: 'none'
NumObservations: 32561
NumTrained: 100
Method: 'LogitBoost'
LearnerNames: {'Tree'}
ReasonForTermination: 'Terminated normally after completing the requested number of training cycles.'
FitInfo: [100x1 double]
FitInfoDescription: {2x1 cell}
Properties, Methods
Mdl is a ClassificationEnsemble model. Some notable characteristics of Mdl are:
Because two classes are represented in the data, LogitBoost is the ensemble aggregation algorithm.
Because the ensemble aggregation method is a boosting algorithm, classification trees that allow a maximum of 10 splits compose the ensemble.
One hundred trees compose the ensemble.
Use the classification ensemble to predict the labels of a random set of five observations from the data. Compare the predicted labels with their true values.
rng(1) % For reproducibility [pX,pIdx] = datasample(adultdata,5); label = predict(Mdl1,pX); table(label,adultdata.salary(pIdx),'VariableNames',{'Predicted','Truth'})
ans=5×2 table
Predicted Truth
_________ _____
<=50K <=50K
<=50K <=50K
<=50K <=50K
<=50K <=50K
<=50K <=50K
Train a new ensemble using age and education only.
Mdl2 = fitcensemble(adultdata,'salary ~ age + education');Compare the resubstitution losses between Mdl1 and Mdl2.
rsLoss1 = resubLoss(Mdl1)
rsLoss1 = 0.1058
rsLoss2 = resubLoss(Mdl2)
rsLoss2 = 0.2037
The in-sample misclassification rate for the ensemble that uses all predictors is lower.
Speed Up Training by Binning Numeric Predictor Values
Train an ensemble of boosted classification trees by using fitcensemble. Reduce training time by specifying the 'NumBins' name-value pair argument to bin numeric predictors. This argument is valid only when fitcensemble uses a tree learner. After training, you can reproduce binned predictor data by using the BinEdges property of the trained model and the discretize function.
Generate a sample data set.
rng('default') % For reproducibility N = 1e6; X = [mvnrnd([-1 -1],eye(2),N); mvnrnd([1 1],eye(2),N)]; y = [zeros(N,1); ones(N,1)];
Visualize the data set.
figure scatter(X(1:N,1),X(1:N,2),'Marker','.','MarkerEdgeAlpha',0.01) hold on scatter(X(N+1:2*N,1),X(N+1:2*N,2),'Marker','.','MarkerEdgeAlpha',0.01)
Train an ensemble of boosted classification trees using adaptive logistic regression (LogitBoost, the default for binary classification). Time the function for comparison purposes.
tic Mdl1 = fitcensemble(X,y); toc
Elapsed time is 478.988422 seconds.
Speed up training by using the 'NumBins' name-value pair argument. If you specify the 'NumBins' value as a positive integer scalar, then the software bins every numeric predictor into a specified number of equiprobable bins, and then grows trees on the bin indices instead of the original data. The software does not bin categorical predictors.
tic
Mdl2 = fitcensemble(X,y,'NumBins',50);
tocElapsed time is 165.598434 seconds.
The process is about three times faster when you use binned data instead of the original data. Note that the elapsed time can vary depending on your operating system.
Compare the classification errors by resubstitution.
rsLoss1 = resubLoss(Mdl1)
rsLoss1 = 0.0788
rsLoss2 = resubLoss(Mdl2)
rsLoss2 = 0.0788
In this example, binning predictor values reduces training time without loss of accuracy. In general, when you have a large data set like the one in this example, using the binning option speeds up training but causes a potential decrease in accuracy. If you want to reduce training time further, specify a smaller number of bins.
Reproduce binned predictor data by using the BinEdges property of the trained model and the discretize function.
X = Mdl2.X; % Predictor data Xbinned = zeros(size(X)); edges = Mdl2.BinEdges; % Find indices of binned predictors. idxNumeric = find(~cellfun(@isempty,edges)); if iscolumn(idxNumeric) idxNumeric = idxNumeric'; end for j = idxNumeric x = X(:,j); % Convert x to array if x is a table. if istable(x) x = table2array(x); end % Group x into bins by using the discretize function. xbinned = discretize(x,[-inf; edges{j}; inf]); Xbinned(:,j) = xbinned; end
Xbinned contains the bin indices, ranging from 1 to the number of bins, for numeric predictors. Xbinned values are 0 for categorical predictors. If X contains NaNs, then the corresponding Xbinned values are NaNs.
Estimate Generalization Error of Boosting Ensemble
Estimate the generalization error of ensemble of boosted classification trees.
Load the ionosphere data set.
load ionosphereCross-validate an ensemble of classification trees using AdaBoostM1 and 10-fold cross-validation. Specify that each tree should be split a maximum of five times using a decision tree template.
rng(5); % For reproducibility t = templateTree('MaxNumSplits',5); Mdl = fitcensemble(X,Y,'Method','AdaBoostM1','Learners',t,'CrossVal','on');
Mdl is a ClassificationPartitionedEnsemble model.
Plot the cumulative, 10-fold cross-validated, misclassification rate. Display the estimated generalization error of the ensemble.
kflc = kfoldLoss(Mdl,'Mode','cumulative'); figure; plot(kflc); ylabel('10-fold Misclassification rate'); xlabel('Learning cycle');
estGenError = kflc(end)
estGenError = 0.0769
kfoldLoss returns the generalization error by default. However, plotting the cumulative loss allows you to monitor how the loss changes as weak learners accumulate in the ensemble.
The ensemble achieves a misclassification rate of around 0.06 after accumulating about 50 weak learners. Then, the misclassification rate increase slightly as more weak learners enter the ensemble.
If you are satisfied with the generalization error of the ensemble, then, to create a predictive model, train the ensemble again using all of the settings except cross-validation. However, it is good practice to tune hyperparameters, such as the maximum number of decision splits per tree and the number of learning cycles.
Optimize Classification Ensemble
Optimize hyperparameters automatically using fitcensemble.
Load the ionosphere data set.
load ionosphereYou can find hyperparameters that minimize five-fold cross-validation loss by using automatic hyperparameter optimization.
Mdl = fitcensemble(X,Y,'OptimizeHyperparameters','auto')
In this example, for reproducibility, set the random seed and use the 'expected-improvement-plus' acquisition function. Also, for reproducibility of random forest algorithm, specify the 'Reproducible' name-value pair argument as true for tree learners.
rng('default') t = templateTree('Reproducible',true); Mdl = fitcensemble(X,Y,'OptimizeHyperparameters','auto','Learners',t, ... 'HyperparameterOptimizationOptions',struct('AcquisitionFunctionName','expected-improvement-plus'))
|===================================================================================================================================| | Iter | Eval | Objective | Objective | BestSoFar | BestSoFar | Method | NumLearningC-| LearnRate | MinLeafSize | | | result | | runtime | (observed) | (estim.) | | ycles | | | |===================================================================================================================================| | 1 | Best | 0.10256 | 2.8201 | 0.10256 | 0.10256 | RUSBoost | 11 | 0.010199 | 17 | | 2 | Best | 0.082621 | 6.3089 | 0.082621 | 0.083414 | LogitBoost | 206 | 0.96537 | 33 | | 3 | Accept | 0.099715 | 4.0004 | 0.082621 | 0.082624 | AdaBoostM1 | 130 | 0.0072814 | 2 | | 4 | Best | 0.068376 | 1.5887 | 0.068376 | 0.068395 | Bag | 25 | - | 5 | | 5 | Best | 0.059829 | 1.7618 | 0.059829 | 0.062829 | LogitBoost | 58 | 0.19016 | 5 | | 6 | Accept | 0.068376 | 1.6662 | 0.059829 | 0.065561 | LogitBoost | 58 | 0.10005 | 5 | | 7 | Accept | 0.088319 | 13.07 | 0.059829 | 0.065786 | LogitBoost | 494 | 0.014474 | 3 | | 8 | Accept | 0.065527 | 0.79673 | 0.059829 | 0.065894 | LogitBoost | 26 | 0.75515 | 8 | | 9 | Accept | 0.15385 | 0.93354 | 0.059829 | 0.061156 | LogitBoost | 32 | 0.0010037 | 59 | | 10 | Accept | 0.059829 | 3.8828 | 0.059829 | 0.059731 | LogitBoost | 143 | 0.44428 | 1 | | 11 | Accept | 0.35897 | 2.3272 | 0.059829 | 0.059826 | Bag | 54 | - | 175 | | 12 | Accept | 0.068376 | 0.53634 | 0.059829 | 0.059825 | Bag | 10 | - | 1 | | 13 | Accept | 0.12251 | 9.5155 | 0.059829 | 0.059826 | AdaBoostM1 | 442 | 0.57897 | 102 | | 14 | Accept | 0.11966 | 4.9323 | 0.059829 | 0.059827 | RUSBoost | 95 | 0.80822 | 1 | | 15 | Accept | 0.062678 | 4.2429 | 0.059829 | 0.059826 | GentleBoost | 156 | 0.99502 | 1 | | 16 | Accept | 0.065527 | 3.0688 | 0.059829 | 0.059824 | GentleBoost | 115 | 0.99693 | 13 | | 17 | Best | 0.05698 | 1.659 | 0.05698 | 0.056997 | GentleBoost | 60 | 0.0010045 | 3 | | 18 | Accept | 0.13675 | 2.0647 | 0.05698 | 0.057002 | GentleBoost | 86 | 0.0010263 | 108 | | 19 | Accept | 0.062678 | 2.4037 | 0.05698 | 0.05703 | GentleBoost | 88 | 0.6344 | 4 | | 20 | Accept | 0.065527 | 1.029 | 0.05698 | 0.057228 | GentleBoost | 35 | 0.0010155 | 1 | |===================================================================================================================================| | Iter | Eval | Objective | Objective | BestSoFar | BestSoFar | Method | NumLearningC-| LearnRate | MinLeafSize | | | result | | runtime | (observed) | (estim.) | | ycles | | | |===================================================================================================================================| | 21 | Accept | 0.079772 | 0.44308 | 0.05698 | 0.057214 | LogitBoost | 11 | 0.9796 | 2 | | 22 | Accept | 0.065527 | 21.191 | 0.05698 | 0.057523 | Bag | 499 | - | 1 | | 23 | Accept | 0.068376 | 20.294 | 0.05698 | 0.057671 | Bag | 494 | - | 2 | | 24 | Accept | 0.64103 | 1.2793 | 0.05698 | 0.057468 | RUSBoost | 30 | 0.088421 | 174 | | 25 | Accept | 0.088319 | 0.53606 | 0.05698 | 0.057456 | RUSBoost | 10 | 0.010292 | 5 | | 26 | Accept | 0.074074 | 0.36802 | 0.05698 | 0.05753 | AdaBoostM1 | 11 | 0.14192 | 13 | | 27 | Accept | 0.099715 | 12.133 | 0.05698 | 0.057646 | AdaBoostM1 | 498 | 0.0010096 | 6 | | 28 | Accept | 0.079772 | 10.877 | 0.05698 | 0.057886 | AdaBoostM1 | 474 | 0.030547 | 31 | | 29 | Accept | 0.068376 | 12.326 | 0.05698 | 0.061326 | GentleBoost | 493 | 0.36142 | 2 | | 30 | Accept | 0.065527 | 0.3945 | 0.05698 | 0.061165 | LogitBoost | 11 | 0.71408 | 16 |
__________________________________________________________
Optimization completed.
MaxObjectiveEvaluations of 30 reached.
Total function evaluations: 30
Total elapsed time: 165.9329 seconds
Total objective function evaluation time: 148.4504
Best observed feasible point:
Method NumLearningCycles LearnRate MinLeafSize
___________ _________________ _________ ___________
GentleBoost 60 0.0010045 3
Observed objective function value = 0.05698
Estimated objective function value = 0.061165
Function evaluation time = 1.659
Best estimated feasible point (according to models):
Method NumLearningCycles LearnRate MinLeafSize
___________ _________________ _________ ___________
GentleBoost 60 0.0010045 3
Estimated objective function value = 0.061165
Estimated function evaluation time = 1.6503
Mdl =
ClassificationEnsemble
ResponseName: 'Y'
CategoricalPredictors: []
ClassNames: {'b' 'g'}
ScoreTransform: 'none'
NumObservations: 351
HyperparameterOptimizationResults: [1×1 BayesianOptimization]
NumTrained: 60
Method: 'GentleBoost'
LearnerNames: {'Tree'}
ReasonForTermination: 'Terminated normally after completing the requested number of training cycles.'
FitInfo: [60×1 double]
FitInfoDescription: {2×1 cell}
Properties, Methods
The optimization searched over the ensemble aggregation methods for binary classification, over NumLearningCycles, over the LearnRate for applicable methods, and over the tree learner MinLeafSize. The output is the ensemble classifier with the minimum estimated cross-validation loss.
Optimize Classification Ensemble Using Cross-Validation
One way to create an ensemble of boosted classification trees that has satisfactory predictive performance is by tuning the decision tree complexity level using cross-validation. While searching for an optimal complexity level, tune the learning rate to minimize the number of learning cycles.
This example manually finds optimal parameters by using the cross-validation option (the 'KFold' name-value pair argument) and the kfoldLoss function. Alternatively, you can use the 'OptimizeHyperparameters' name-value pair argument to optimize hyperparameters automatically. See Optimize Classification Ensemble.
Load the ionosphere data set.
load ionosphereTo search for the optimal tree-complexity level:
Cross-validate a set of ensembles. Exponentially increase the tree-complexity level for subsequent ensembles from decision stump (one split) to at most n - 1 splits. n is the sample size. Also, vary the learning rate for each ensemble between 0.1 to 1.
Estimate the cross-validated misclassification rate of each ensemble.
For tree-complexity level , , compare the cumulative, cross-validated misclassification rate of the ensembles by plotting them against number of learning cycles. Plot separate curves for each learning rate on the same figure.
Choose the curve that achieves the minimal misclassification rate, and note the corresponding learning cycle and learning rate.
Cross-validate a deep classification tree and a stump. These classification trees serve as benchmarks.
rng(1) % For reproducibility MdlDeep = fitctree(X,Y,'CrossVal','on','MergeLeaves','off', ... 'MinParentSize',1); MdlStump = fitctree(X,Y,'MaxNumSplits',1,'CrossVal','on');
Cross-validate an ensemble of 150 boosted classification trees using 5-fold cross-validation. Using a tree template, vary the maximum number of splits using the values in the sequence . m is such that is no greater than n - 1. For each variant, adjust the learning rate using each value in the set {0.1, 0.25, 0.5, 1};
n = size(X,1); m = floor(log(n - 1)/log(3)); learnRate = [0.1 0.25 0.5 1]; numLR = numel(learnRate); maxNumSplits = 3.^(0:m); numMNS = numel(maxNumSplits); numTrees = 150; Mdl = cell(numMNS,numLR); for k = 1:numLR for j = 1:numMNS t = templateTree('MaxNumSplits',maxNumSplits(j)); Mdl{j,k} = fitcensemble(X,Y,'NumLearningCycles',numTrees,... 'Learners',t,'KFold',5,'LearnRate',learnRate(k)); end end
Estimate the cumulative, cross-validated misclassification rate for each ensemble and the classification trees serving as benchmarks.
kflAll = @(x)kfoldLoss(x,'Mode','cumulative'); errorCell = cellfun(kflAll,Mdl,'Uniform',false); error = reshape(cell2mat(errorCell),[numTrees numel(maxNumSplits) numel(learnRate)]); errorDeep = kfoldLoss(MdlDeep); errorStump = kfoldLoss(MdlStump);
Plot how the cross-validated misclassification rate behaves as the number of trees in the ensemble increases. Plot the curves with respect to learning rate on the same plot, and plot separate plots for varying tree-complexity levels. Choose a subset of tree complexity levels to plot.
mnsPlot = [1 round(numel(maxNumSplits)/2) numel(maxNumSplits)]; figure for k = 1:3 subplot(2,2,k) plot(squeeze(error(:,mnsPlot(k),:)),'LineWidth',2) axis tight hold on h = gca; plot(h.XLim,[errorDeep errorDeep],'-.b','LineWidth',2) plot(h.XLim,[errorStump errorStump],'-.r','LineWidth',2) plot(h.XLim,min(min(error(:,mnsPlot(k),:))).*[1 1],'--k') h.YLim = [0 0.2]; xlabel('Number of trees') ylabel('Cross-validated misclass. rate') title(sprintf('MaxNumSplits = %0.3g', maxNumSplits(mnsPlot(k)))) hold off end hL = legend([cellstr(num2str(learnRate','Learning Rate = %0.2f')); ... 'Deep Tree';'Stump';'Min. misclass. rate']); hL.Position(1) = 0.6;
Each curve contains a minimum cross-validated misclassification rate occurring at the optimal number of trees in the ensemble.
Identify the maximum number of splits, number of trees, and learning rate that yields the lowest misclassification rate overall.
[minErr,minErrIdxLin] = min(error(:));
[idxNumTrees,idxMNS,idxLR] = ind2sub(size(error),minErrIdxLin);
fprintf('\nMin. misclass. rate = %0.5f',minErr)Min. misclass. rate = 0.05128
fprintf('\nOptimal Parameter Values:\nNum. Trees = %d',idxNumTrees);Optimal Parameter Values: Num. Trees = 130
fprintf('\nMaxNumSplits = %d\nLearning Rate = %0.2f\n',... maxNumSplits(idxMNS),learnRate(idxLR))
MaxNumSplits = 9 Learning Rate = 1.00
Create a predictive ensemble based on the optimal hyperparameters and the entire training set.
tFinal = templateTree('MaxNumSplits',maxNumSplits(idxMNS)); MdlFinal = fitcensemble(X,Y,'NumLearningCycles',idxNumTrees,... 'Learners',tFinal,'LearnRate',learnRate(idxLR))
MdlFinal =
ClassificationEnsemble
ResponseName: 'Y'
CategoricalPredictors: []
ClassNames: {'b' 'g'}
ScoreTransform: 'none'
NumObservations: 351
NumTrained: 130
Method: 'LogitBoost'
LearnerNames: {'Tree'}
ReasonForTermination: 'Terminated normally after completing the requested number of training cycles.'
FitInfo: [130×1 double]
FitInfoDescription: {2×1 cell}
Properties, Methods
MdlFinal is a ClassificationEnsemble. To predict whether a radar return is good given predictor data, you can pass the predictor data and MdlFinal to predict.
Instead of searching optimal values manually by using the cross-validation option ('KFold') and the kfoldLoss function, you can use the 'OptimizeHyperparameters' name-value pair argument. When you specify 'OptimizeHyperparameters', the software finds optimal parameters automatically using Bayesian optimization. The optimal values obtained by using 'OptimizeHyperparameters' can be different from those obtained using manual search.
mdl = fitcensemble(X,Y,'OptimizeHyperparameters',{'NumLearningCycles','LearnRate','MaxNumSplits'})
|====================================================================================================================| | Iter | Eval | Objective | Objective | BestSoFar | BestSoFar | NumLearningC-| LearnRate | MaxNumSplits | | | result | | runtime | (observed) | (estim.) | ycles | | | |====================================================================================================================| | 1 | Best | 0.094017 | 3.7194 | 0.094017 | 0.094017 | 137 | 0.001364 | 3 | | 2 | Accept | 0.12251 | 0.66511 | 0.094017 | 0.095735 | 15 | 0.013089 | 144 |
| 3 | Best | 0.065527 | 0.90035 | 0.065527 | 0.067815 | 31 | 0.47201 | 2 | | 4 | Accept | 0.19943 | 8.6107 | 0.065527 | 0.070015 | 340 | 0.92167 | 7 | | 5 | Accept | 0.071225 | 0.90081 | 0.065527 | 0.065583 | 32 | 0.14422 | 2 | | 6 | Accept | 0.099715 | 0.688 | 0.065527 | 0.065573 | 23 | 0.0010566 | 2 | | 7 | Accept | 0.11681 | 0.90799 | 0.065527 | 0.065565 | 28 | 0.0010156 | 259 | | 8 | Accept | 0.17379 | 0.82143 | 0.065527 | 0.065559 | 29 | 0.0013435 | 1 | | 9 | Best | 0.059829 | 0.59677 | 0.059829 | 0.059844 | 18 | 0.87865 | 3 | | 10 | Accept | 0.11111 | 0.40132 | 0.059829 | 0.059843 | 10 | 0.0012112 | 48 | | 11 | Accept | 0.08547 | 0.41121 | 0.059829 | 0.059842 | 10 | 0.62108 | 25 | | 12 | Accept | 0.11681 | 0.41538 | 0.059829 | 0.059841 | 10 | 0.0012154 | 20 | | 13 | Accept | 0.082621 | 0.46504 | 0.059829 | 0.059842 | 10 | 0.55351 | 35 | | 14 | Accept | 0.079772 | 0.46297 | 0.059829 | 0.05984 | 11 | 0.74109 | 74 | | 15 | Accept | 0.088319 | 0.69297 | 0.059829 | 0.05984 | 19 | 0.91106 | 347 | | 16 | Accept | 0.062678 | 0.3637 | 0.059829 | 0.059886 | 10 | 0.97239 | 3 | | 17 | Accept | 0.065527 | 1.9404 | 0.059829 | 0.059887 | 78 | 0.97069 | 3 | | 18 | Accept | 0.065527 | 0.39816 | 0.059829 | 0.062228 | 11 | 0.75051 | 2 | | 19 | Best | 0.054131 | 0.36381 | 0.054131 | 0.059083 | 10 | 0.69072 | 3 | | 20 | Accept | 0.065527 | 0.38429 | 0.054131 | 0.060938 | 10 | 0.64403 | 3 | |====================================================================================================================| | Iter | Eval | Objective | Objective | BestSoFar | BestSoFar | NumLearningC-| LearnRate | MaxNumSplits | | | result | | runtime | (observed) | (estim.) | ycles | | | |====================================================================================================================| | 21 | Accept | 0.079772 | 0.40405 | 0.054131 | 0.060161 | 10 | 0.80548 | 13 | | 22 | Accept | 0.05698 | 0.37983 | 0.054131 | 0.059658 | 10 | 0.56949 | 5 | | 23 | Accept | 0.10826 | 0.36128 | 0.054131 | 0.059244 | 10 | 0.0055133 | 5 | | 24 | Accept | 0.074074 | 0.38056 | 0.054131 | 0.05933 | 10 | 0.92056 | 6 | | 25 | Accept | 0.11966 | 0.35336 | 0.054131 | 0.059132 | 10 | 0.27254 | 1 | | 26 | Accept | 0.065527 | 0.77041 | 0.054131 | 0.059859 | 26 | 0.97412 | 3 | | 27 | Accept | 0.068376 | 0.38116 | 0.054131 | 0.060205 | 10 | 0.82146 | 4 | | 28 | Accept | 0.062678 | 0.47015 | 0.054131 | 0.060713 | 14 | 0.99445 | 3 | | 29 | Accept | 0.11966 | 0.41033 | 0.054131 | 0.060826 | 10 | 0.0012621 | 344 | | 30 | Accept | 0.08547 | 0.45352 | 0.054131 | 0.060771 | 10 | 0.93676 | 187 |
__________________________________________________________
Optimization completed.
MaxObjectiveEvaluations of 30 reached.
Total function evaluations: 30
Total elapsed time: 41.5854 seconds
Total objective function evaluation time: 28.4744
Best observed feasible point:
NumLearningCycles LearnRate MaxNumSplits
_________________ _________ ____________
10 0.69072 3
Observed objective function value = 0.054131
Estimated objective function value = 0.061741
Function evaluation time = 0.36381
Best estimated feasible point (according to models):
NumLearningCycles LearnRate MaxNumSplits
_________________ _________ ____________
14 0.99445 3
Estimated objective function value = 0.060771
Estimated function evaluation time = 0.48009
mdl =
ClassificationEnsemble
ResponseName: 'Y'
CategoricalPredictors: []
ClassNames: {'b' 'g'}
ScoreTransform: 'none'
NumObservations: 351
HyperparameterOptimizationResults: [1×1 BayesianOptimization]
NumTrained: 14
Method: 'LogitBoost'
LearnerNames: {'Tree'}
ReasonForTermination: 'Terminated normally after completing the requested number of training cycles.'
FitInfo: [14×1 double]
FitInfoDescription: {2×1 cell}
Properties, Methods
Input Arguments
Output Arguments
Tips
NumLearningCyclescan vary from a few dozen to a few thousand. Usually, an ensemble with good predictive power requires from a few hundred to a few thousand weak learners. However, you do not have to train an ensemble for that many cycles at once. You can start by growing a few dozen learners, inspect the ensemble performance and then, if necessary, train more weak learners usingresumefor classification problems.Ensemble performance depends on the ensemble setting and the setting of the weak learners. That is, if you specify weak learners with default parameters, then the ensemble can perform poorly. Therefore, like ensemble settings, it is good practice to adjust the parameters of the weak learners using templates, and to choose values that minimize generalization error.
If you specify to resample using
Resample, then it is good practice to resample to entire data set. That is, use the default setting of1forFResample.If the ensemble aggregation method (
Method) is'bag'and:The misclassification cost (
Cost) is highly imbalanced, then, for in-bag samples, the software oversamples unique observations from the class that has a large penalty.The class prior probabilities (
Prior) are highly skewed, the software oversamples unique observations from the class that has a large prior probability.
For smaller sample sizes, these combinations can result in a low relative frequency of out-of-bag observations from the class that has a large penalty or prior probability. Consequently, the estimated out-of-bag error is highly variable and it can be difficult to interpret. To avoid large estimated out-of-bag error variances, particularly for small sample sizes, set a more balanced misclassification cost matrix using
Costor a less skewed prior probability vector usingPrior.Because the order of some input and output arguments correspond to the distinct classes in the training data, it is good practice to specify the class order using the
ClassNamesname-value pair argument.To determine the class order quickly, remove all observations from the training data that are unclassified (that is, have a missing label), obtain and display an array of all the distinct classes, and then specify the array for
ClassNames. For example, suppose the response variable (Y) is a cell array of labels. This code specifies the class order in the variableclassNames.Ycat = categorical(Y); classNames = categories(Ycat)
categoricalassigns<undefined>to unclassified observations andcategoriesexcludes<undefined>from its output. Therefore, if you use this code for cell arrays of labels or similar code for categorical arrays, then you do not have to remove observations with missing labels to obtain a list of the distinct classes.To specify that the class order from lowest-represented label to most-represented, then quickly determine the class order (as in the previous bullet), but arrange the classes in the list by frequency before passing the list to
ClassNames. Following from the previous example, this code specifies the class order from lowest- to most-represented inclassNamesLH.Ycat = categorical(Y); classNames = categories(Ycat); freq = countcats(Ycat); [~,idx] = sort(freq); classNamesLH = classNames(idx);
After training a model, you can generate C/C++ code that predicts labels for new data. Generating C/C++ code requires MATLAB Coder™. For details, see Introduction to Code Generation.
Algorithms
For details of ensemble aggregation algorithms, see Ensemble Algorithms.
If you set
Methodto be a boosting algorithm andLearnersto be decision trees, then the software grows shallow decision trees by default. You can adjust tree depth by specifying theMaxNumSplits,MinLeafSize, andMinParentSizename-value pair arguments usingtemplateTree.If you specify the
Cost,Prior, andWeightsname-value arguments, the output model object stores the specified values in theCost,Prior, andWproperties, respectively. TheCostproperty stores the user-specified cost matrix (C) as is. ThePriorandWproperties store the prior probabilities and observation weights, respectively, after normalization. For model training, the software updates the prior probabilities and observation weights to incorporate the penalties described in the cost matrix. For details, see Misclassification Cost Matrix, Prior Probabilities, and Observation Weights.For bagging (
'Method','Bag'),fitcensemblegenerates in-bag samples by oversampling classes with large misclassification costs and undersampling classes with small misclassification costs. Consequently, out-of-bag samples have fewer observations from classes with large misclassification costs and more observations from classes with small misclassification costs. If you train a classification ensemble using a small data set and a highly skewed cost matrix, then the number of out-of-bag observations per class can be low. Therefore, the estimated out-of-bag error can have a large variance and can be difficult to interpret. The same phenomenon can occur for classes with large prior probabilities.For the RUSBoost ensemble aggregation method (
'Method','RUSBoost'), the name-value pair argumentRatioToSmallestspecifies the sampling proportion for each class with respect to the lowest-represented class. For example, suppose that there are two classes in the training data: A and B. A has 100 observations and B has 10 observations. Suppose also that the lowest-represented class hasmobservations in the training data.If you set
'RatioToSmallest',2, thens*m2*10=20. Consequently,fitcensembletrains every learner using 20 observations from class A and 20 observations from class B. If you set'RatioToSmallest',[2 2], then you obtain the same result.If you set
'RatioToSmallest',[2,1], thens1*m2*10=20ands2*m1*10=10. Consequently,fitcensembletrains every learner using 20 observations from class A and 10 observations from class B.
For dual-core systems and above,
fitcensembleparallelizes training using Intel Threading Building Blocks (TBB). For details on Intel TBB, see https://software.intel.com/content/www/us/en/develop/tools/oneapi/components/onetbb.html.
References
[1] Breiman, L. “Bagging Predictors.” Machine Learning. Vol. 26, pp. 123–140, 1996.
[2] Breiman, L. “Random Forests.” Machine Learning. Vol. 45, pp. 5–32, 2001.
[3] Freund, Y. “A more robust boosting algorithm.” arXiv:0905.2138v1, 2009.
[4] Freund, Y. and R. E. Schapire. “A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting.” J. of Computer and System Sciences, Vol. 55, pp. 119–139, 1997.
[5] Friedman, J. “Greedy function approximation: A gradient boosting machine.” Annals of Statistics, Vol. 29, No. 5, pp. 1189–1232, 2001.
[6] Friedman, J., T. Hastie, and R. Tibshirani. “Additive logistic regression: A statistical view of boosting.” Annals of Statistics, Vol. 28, No. 2, pp. 337–407, 2000.
[7] Hastie, T., R. Tibshirani, and J. Friedman. The Elements of Statistical Learning section edition, Springer, New York, 2008.
[8] Ho, T. K. “The random subspace method for constructing decision forests.” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, No. 8, pp. 832–844, 1998.
[9] Schapire, R. E., Y. Freund, P. Bartlett, and W.S. Lee. “Boosting the margin: A new explanation for the effectiveness of voting methods.” Annals of Statistics, Vol. 26, No. 5, pp. 1651–1686, 1998.
[10] Seiffert, C., T. Khoshgoftaar, J. Hulse, and A. Napolitano. “RUSBoost: Improving classification performance when training data is skewed.” 19th International Conference on Pattern Recognition, pp. 1–4, 2008.
[11] Warmuth, M., J. Liao, and G. Ratsch. “Totally corrective boosting algorithms that maximize the margin.” Proc. 23rd Int’l. Conf. on Machine Learning, ACM, New York, pp. 1001–1008, 2006.
Extended Capabilities
Version History
You can also select a web site from the following list:
Americas
- América Latina (Español)
- Canada (English)
- United States (English)
Europe
- Belgium (English)
- Denmark (English)
- Deutschland (Deutsch)
- España (Español)
- Finland (English)
- France (Français)
- Ireland (English)
- Italia (Italiano)
- Luxembourg (English)
- Netherlands (English)
- Norway (English)
- Österreich (Deutsch)
- Portugal (English)
- Sweden (English)
- Switzerland
- United Kingdom (English)