Fit ensemble of learners for classification
returns the trained classification ensemble model object
(Mdl
= fitcensemble(Tbl
,ResponseVarName
)Mdl
) that contains the results of boosting 100
classification trees and the predictor and response data in the table
Tbl
. ResponseVarName
is the name of
the response variable in Tbl
. By default,
fitcensemble
uses LogitBoost for binary classification
and AdaBoostM2 for multiclass classification.
applies Mdl
= fitcensemble(Tbl
,formula
)formula
to fit the model to the predictor and
response data in the table Tbl
. formula
is
an explanatory model of the response and a subset of predictor variables in
Tbl
used to fit Mdl
. For example,
'Y~X1+X2+X3'
fits the response variable
Tbl.Y
as a function of the predictor variables
Tbl.X1
, Tbl.X2
, and
Tbl.X3
.
uses additional options specified by one or more Mdl
= fitcensemble(___,Name,Value
)Name,Value
pair arguments and any of the input arguments in the previous syntaxes. For
example, you can specify the number of learning cycles, the ensemble aggregation
method, or to implement 10-fold cross-validation.
Create a predictive classification ensemble using all available predictor variables in the data. Then, train another ensemble using fewer predictors. Compare the in-sample predictive accuracies of the ensembles.
Load the census1994
data set.
load census1994
Train an ensemble of classification models using the entire data set and default options.
Mdl1 = fitcensemble(adultdata,'salary')
Mdl1 = ClassificationEnsemble PredictorNames: {1x14 cell} ResponseName: 'salary' CategoricalPredictors: [2 4 6 7 8 9 10 14] ClassNames: [<=50K >50K] ScoreTransform: 'none' NumObservations: 32561 NumTrained: 100 Method: 'LogitBoost' LearnerNames: {'Tree'} ReasonForTermination: 'Terminated normally after completing the requested number of training cycles.' FitInfo: [100x1 double] FitInfoDescription: {2x1 cell} Properties, Methods
Mdl
is a ClassificationEnsemble
model. Some notable characteristics of Mdl
are:
Because two classes are represented in the data, LogitBoost is the ensemble aggregation algorithm.
Because the ensemble aggregation method is a boosting algorithm, classification trees that allow a maximum of 10 splits compose the ensemble.
One hundred trees compose the ensemble.
Use the classification ensemble to predict the labels of a random set of five observations from the data. Compare the predicted labels with their true values.
rng(1) % For reproducibility [pX,pIdx] = datasample(adultdata,5); label = predict(Mdl1,pX); table(label,adultdata.salary(pIdx),'VariableNames',{'Predicted','Truth'})
ans=5×2 table
Predicted Truth
_________ _____
<=50K <=50K
<=50K <=50K
<=50K <=50K
<=50K <=50K
<=50K <=50K
Train a new ensemble using age
and education
only.
Mdl2 = fitcensemble(adultdata,'salary ~ age + education');
Compare the resubstitution losses between Mdl1
and Mdl2
.
rsLoss1 = resubLoss(Mdl1)
rsLoss1 = 0.1058
rsLoss2 = resubLoss(Mdl2)
rsLoss2 = 0.2037
The in-sample misclassification rate for the ensemble that uses all predictors is lower.
Train an ensemble of boosted classification trees by using fitcensemble
. Reduce training time by specifying the 'NumBins'
name-value pair argument to bin numeric predictors. This argument is valid only when fitcensemble
uses a tree learner. After training, you can reproduce binned predictor data by using the BinEdges
property of the trained model and the discretize
function.
Generate a sample data set.
rng('default') % For reproducibility N = 1e6; X = [mvnrnd([-1 -1],eye(2),N); mvnrnd([1 1],eye(2),N)]; y = [zeros(N,1); ones(N,1)];
Visualize the data set.
figure scatter(X(1:N,1),X(1:N,2),'Marker','.','MarkerEdgeAlpha',0.01) hold on scatter(X(N+1:2*N,1),X(N+1:2*N,2),'Marker','.','MarkerEdgeAlpha',0.01)
Train an ensemble of boosted classification trees using adaptive logistic regression (LogitBoost
, the default for binary classification). Time the function for comparison purposes.
tic Mdl1 = fitcensemble(X,y); toc
Elapsed time is 478.988422 seconds.
Speed up training by using the 'NumBins'
name-value pair argument. If you specify the 'NumBins'
value as a positive integer scalar, then the software bins every numeric predictor into a specified number of equiprobable bins, and then grows trees on the bin indices instead of the original data. The software does not bin categorical predictors.
tic
Mdl2 = fitcensemble(X,y,'NumBins',50);
toc
Elapsed time is 165.598434 seconds.
The process is about three times faster when you use binned data instead of the original data. Note that the elapsed time can vary depending on your operating system.
Compare the classification errors by resubstitution.
rsLoss1 = resubLoss(Mdl1)
rsLoss1 = 0.0788
rsLoss2 = resubLoss(Mdl2)
rsLoss2 = 0.0788
In this example, binning predictor values reduces training time without loss of accuracy. In general, when you have a large data set like the one in this example, using the binning option speeds up training but causes a potential decrease in accuracy. If you want to reduce training time further, specify a smaller number of bins.
Reproduce binned predictor data by using the BinEdges
property of the trained model and the discretize
function.
X = Mdl2.X; % Predictor data Xbinned = zeros(size(X)); edges = Mdl2.BinEdges; % Find indices of binned predictors. idxNumeric = find(~cellfun(@isempty,edges)); if iscolumn(idxNumeric) idxNumeric = idxNumeric'; end for j = idxNumeric x = X(:,j); % Convert x to array if x is a table. if istable(x) x = table2array(x); end % Group x into bins by using the discretize function. xbinned = discretize(x,[-inf; edges{j}; inf]); Xbinned(:,j) = xbinned; end
Xbinned
contains the bin indices, ranging from 1 to the number of bins, for numeric predictors. Xbinned
values are 0
for categorical predictors. If X
contains NaN
s, then the corresponding Xbinned
values are NaN
s.
Estimate the generalization error of ensemble of boosted classification trees.
Load the ionosphere
data set.
load ionosphere
Cross-validate an ensemble of classification trees using AdaBoostM1 and 10-fold cross-validation. Specify that each tree should be split a maximum of five times using a decision tree template.
rng(5); % For reproducibility t = templateTree('MaxNumSplits',5); Mdl = fitcensemble(X,Y,'Method','AdaBoostM1','Learners',t,'CrossVal','on');
Mdl
is a ClassificationPartitionedEnsemble
model.
Plot the cumulative, 10-fold cross-validated, misclassification rate. Display the estimated generalization error of the ensemble.
kflc = kfoldLoss(Mdl,'Mode','cumulative'); figure; plot(kflc); ylabel('10-fold Misclassification rate'); xlabel('Learning cycle');
estGenError = kflc(end)
estGenError = 0.0769
kfoldLoss
returns the generalization error by default. However, plotting the cumulative loss allows you to monitor how the loss changes as weak learners accumulate in the ensemble.
The ensemble achieves a misclassification rate of around 0.06 after accumulating about 50 weak learners. Then, the misclassification rate increase slightly as more weak learners enter the ensemble.
If you are satisfied with the generalization error of the ensemble, then, to create a predictive model, train the ensemble again using all of the settings except cross-validation. However, it is good practice to tune hyperparameters, such as the maximum number of decision splits per tree and the number of learning cycles.
Optimize hyperparameters automatically using fitcensemble
.
Load the ionosphere
data set.
load ionosphere
You can find hyperparameters that minimize five-fold cross-validation loss by using automatic hyperparameter optimization.
Mdl = fitcensemble(X,Y,'OptimizeHyperparameters','auto')
In this example, for reproducibility, set the random seed and use the 'expected-improvement-plus'
acquisition function. Also, for reproducibility of random forest algorithm, specify the 'Reproducible'
name-value pair argument as true
for tree learners.
rng('default') t = templateTree('Reproducible',true); Mdl = fitcensemble(X,Y,'OptimizeHyperparameters','auto','Learners',t, ... 'HyperparameterOptimizationOptions',struct('AcquisitionFunctionName','expected-improvement-plus'))
|===================================================================================================================================| | Iter | Eval | Objective | Objective | BestSoFar | BestSoFar | Method | NumLearningC-| LearnRate | MinLeafSize | | | result | | runtime | (observed) | (estim.) | | ycles | | | |===================================================================================================================================| | 1 | Best | 0.10256 | 1.6978 | 0.10256 | 0.10256 | RUSBoost | 11 | 0.010199 | 17 |
| 2 | Best | 0.062678 | 9.4354 | 0.062678 | 0.064264 | LogitBoost | 206 | 0.96537 | 33 |
| 3 | Accept | 0.099715 | 7.614 | 0.062678 | 0.062688 | AdaBoostM1 | 130 | 0.0072814 | 2 |
| 4 | Accept | 0.068376 | 1.6045 | 0.062678 | 0.062681 | Bag | 25 | - | 5 |
| 5 | Accept | 0.065527 | 20.359 | 0.062678 | 0.062699 | LogitBoost | 447 | 0.5405 | 13 |
| 6 | Accept | 0.074074 | 7.1054 | 0.062678 | 0.0627 | GentleBoost | 157 | 0.60495 | 108 |
| 7 | Accept | 0.082621 | 0.9688 | 0.062678 | 0.064102 | GentleBoost | 19 | 0.0010515 | 42 |
| 8 | Accept | 0.17379 | 0.49564 | 0.062678 | 0.06268 | LogitBoost | 10 | 0.001079 | 149 |
| 9 | Accept | 0.076923 | 21.003 | 0.062678 | 0.062676 | GentleBoost | 468 | 0.035181 | 2 |
| 10 | Accept | 0.068376 | 13.575 | 0.062678 | 0.062676 | AdaBoostM1 | 221 | 0.99976 | 7 |
| 11 | Accept | 0.10541 | 3.6394 | 0.062678 | 0.062676 | RUSBoost | 59 | 0.99629 | 31 |
| 12 | Accept | 0.068376 | 3.3423 | 0.062678 | 0.062674 | AdaBoostM1 | 53 | 0.20568 | 26 |
| 13 | Accept | 0.096866 | 1.6005 | 0.062678 | 0.062672 | RUSBoost | 22 | 0.0010042 | 2 |
| 14 | Accept | 0.071225 | 1.201 | 0.062678 | 0.062688 | LogitBoost | 23 | 0.99624 | 1 |
| 15 | Accept | 0.082621 | 0.87944 | 0.062678 | 0.062687 | AdaBoostM1 | 11 | 0.95241 | 2 |
| 16 | Accept | 0.079772 | 29.788 | 0.062678 | 0.062679 | AdaBoostM1 | 486 | 0.23903 | 35 |
| 17 | Accept | 0.35897 | 23.651 | 0.062678 | 0.06267 | Bag | 499 | - | 169 |
| 18 | Accept | 0.074074 | 0.653 | 0.062678 | 0.062674 | Bag | 10 | - | 1 |
| 19 | Accept | 0.088319 | 32.811 | 0.062678 | 0.062674 | RUSBoost | 498 | 0.0010437 | 3 |
| 20 | Accept | 0.068376 | 6.1279 | 0.062678 | 0.062673 | GentleBoost | 130 | 0.0010021 | 3 |
|===================================================================================================================================| | Iter | Eval | Objective | Objective | BestSoFar | BestSoFar | Method | NumLearningC-| LearnRate | MinLeafSize | | | result | | runtime | (observed) | (estim.) | | ycles | | | |===================================================================================================================================| | 21 | Accept | 0.17379 | 22.601 | 0.062678 | 0.06271 | LogitBoost | 496 | 0.0010096 | 146 |
| 22 | Accept | 0.071225 | 2.9727 | 0.062678 | 0.062713 | GentleBoost | 71 | 0.91141 | 9 |
| 23 | Accept | 0.64103 | 1.1288 | 0.062678 | 0.062706 | RUSBoost | 20 | 0.0012846 | 173 |
| 24 | Accept | 0.11111 | 1.537 | 0.062678 | 0.062697 | RUSBoost | 24 | 0.96694 | 6 |
| 25 | Accept | 0.17379 | 5.5632 | 0.062678 | 0.062686 | LogitBoost | 136 | 0.001 | 3 |
| 26 | Accept | 0.35897 | 8.0556 | 0.062678 | 0.062686 | AdaBoostM1 | 156 | 0.003243 | 174 |
| 27 | Accept | 0.065527 | 1.0791 | 0.062678 | 0.062686 | Bag | 21 | - | 2 |
| 28 | Accept | 0.17379 | 1.7562 | 0.062678 | 0.062689 | LogitBoost | 42 | 0.0010283 | 21 |
| 29 | Accept | 0.074074 | 4.3825 | 0.062678 | 0.062689 | GentleBoost | 108 | 0.0010055 | 173 |
| 30 | Accept | 0.065527 | 1.4893 | 0.062678 | 0.062689 | LogitBoost | 32 | 0.97832 | 4 |
__________________________________________________________ Optimization completed. MaxObjectiveEvaluations of 30 reached. Total function evaluations: 30 Total elapsed time: 278.5509 seconds. Total objective function evaluation time: 238.1176 Best observed feasible point: Method NumLearningCycles LearnRate MinLeafSize __________ _________________ _________ ___________ LogitBoost 206 0.96537 33 Observed objective function value = 0.062678 Estimated objective function value = 0.062689 Function evaluation time = 9.4354 Best estimated feasible point (according to models): Method NumLearningCycles LearnRate MinLeafSize __________ _________________ _________ ___________ LogitBoost 206 0.96537 33 Estimated objective function value = 0.062689 Estimated function evaluation time = 9.4324
Mdl = ClassificationEnsemble ResponseName: 'Y' CategoricalPredictors: [] ClassNames: {'b' 'g'} ScoreTransform: 'none' NumObservations: 351 HyperparameterOptimizationResults: [1×1 BayesianOptimization] NumTrained: 206 Method: 'LogitBoost' LearnerNames: {'Tree'} ReasonForTermination: 'Terminated normally after completing the requested number of training cycles.' FitInfo: [206×1 double] FitInfoDescription: {2×1 cell} Properties, Methods
The optimization searched over the ensemble aggregation methods for binary classification, over NumLearningCycles
, over the LearnRate
for applicable methods, and over the tree learner MinLeafSize
. The output is the ensemble classifier with the minimum estimated cross-validation loss.
One way to create an ensemble of boosted classification trees that has satisfactory predictive performance is by tuning the decision tree complexity level using cross-validation. While searching for an optimal complexity level, tune the learning rate to minimize the number of learning cycles.
This example manually finds optimal parameters by using the cross-validation option (the 'KFold'
name-value pair argument) and the kfoldLoss
function. Alternatively, you can use the 'OptimizeHyperparameters'
name-value pair argument to optimize hyperparameters automatically. See Optimize Classification Ensemble.
Load the ionosphere
data set.
load ionosphere
To search for the optimal tree-complexity level:
Cross-validate a set of ensembles. Exponentially increase the tree-complexity level for subsequent ensembles from decision stump (one split) to at most n - 1 splits. n is the sample size. Also, vary the learning rate for each ensemble between 0.1 to 1.
Estimate the cross-validated misclassification rate of each ensemble.
For tree-complexity level , , compare the cumulative, cross-validated misclassification rate of the ensembles by plotting them against number of learning cycles. Plot separate curves for each learning rate on the same figure.
Choose the curve that achieves the minimal misclassification rate, and note the corresponding learning cycle and learning rate.
Cross-validate a deep classification tree and a stump. These classification trees serve as benchmarks.
rng(1) % For reproducibility MdlDeep = fitctree(X,Y,'CrossVal','on','MergeLeaves','off', ... 'MinParentSize',1); MdlStump = fitctree(X,Y,'MaxNumSplits',1,'CrossVal','on');
Cross-validate an ensemble of 150 boosted classification trees using 5-fold cross-validation. Using a tree template, vary the maximum number of splits using the values in the sequence . m is such that is no greater than n - 1. For each variant, adjust the learning rate using each value in the set {0.1, 0.25, 0.5, 1};
n = size(X,1); m = floor(log(n - 1)/log(3)); learnRate = [0.1 0.25 0.5 1]; numLR = numel(learnRate); maxNumSplits = 3.^(0:m); numMNS = numel(maxNumSplits); numTrees = 150; Mdl = cell(numMNS,numLR); for k = 1:numLR for j = 1:numMNS t = templateTree('MaxNumSplits',maxNumSplits(j)); Mdl{j,k} = fitcensemble(X,Y,'NumLearningCycles',numTrees,... 'Learners',t,'KFold',5,'LearnRate',learnRate(k)); end end
Estimate the cumulative, cross-validated misclassification rate for each ensemble and the classification trees serving as benchmarks.
kflAll = @(x)kfoldLoss(x,'Mode','cumulative'); errorCell = cellfun(kflAll,Mdl,'Uniform',false); error = reshape(cell2mat(errorCell),[numTrees numel(maxNumSplits) numel(learnRate)]); errorDeep = kfoldLoss(MdlDeep); errorStump = kfoldLoss(MdlStump);
Plot how the cross-validated misclassification rate behaves as the number of trees in the ensemble increases. Plot the curves with respect to learning rate on the same plot, and plot separate plots for varying tree-complexity levels. Choose a subset of tree complexity levels to plot.
mnsPlot = [1 round(numel(maxNumSplits)/2) numel(maxNumSplits)]; figure for k = 1:3 subplot(2,2,k) plot(squeeze(error(:,mnsPlot(k),:)),'LineWidth',2) axis tight hold on h = gca; plot(h.XLim,[errorDeep errorDeep],'-.b','LineWidth',2) plot(h.XLim,[errorStump errorStump],'-.r','LineWidth',2) plot(h.XLim,min(min(error(:,mnsPlot(k),:))).*[1 1],'--k') h.YLim = [0 0.2]; xlabel('Number of trees') ylabel('Cross-validated misclass. rate') title(sprintf('MaxNumSplits = %0.3g', maxNumSplits(mnsPlot(k)))) hold off end hL = legend([cellstr(num2str(learnRate','Learning Rate = %0.2f')); ... 'Deep Tree';'Stump';'Min. misclass. rate']); hL.Position(1) = 0.6;
Each curve contains a minimum cross-validated misclassification rate occurring at the optimal number of trees in the ensemble.
Identify the maximum number of splits, number of trees, and learning rate that yields the lowest misclassification rate overall.
[minErr,minErrIdxLin] = min(error(:));
[idxNumTrees,idxMNS,idxLR] = ind2sub(size(error),minErrIdxLin);
fprintf('\nMin. misclass. rate = %0.5f',minErr)
Min. misclass. rate = 0.05413
fprintf('\nOptimal Parameter Values:\nNum. Trees = %d',idxNumTrees);
Optimal Parameter Values: Num. Trees = 47
fprintf('\nMaxNumSplits = %d\nLearning Rate = %0.2f\n',... maxNumSplits(idxMNS),learnRate(idxLR))
MaxNumSplits = 3 Learning Rate = 0.25
Create a predictive ensemble based on the optimal hyperparameters and the entire training set.
tFinal = templateTree('MaxNumSplits',maxNumSplits(idxMNS)); MdlFinal = fitcensemble(X,Y,'NumLearningCycles',idxNumTrees,... 'Learners',tFinal,'LearnRate',learnRate(idxLR))
MdlFinal = ClassificationEnsemble ResponseName: 'Y' CategoricalPredictors: [] ClassNames: {'b' 'g'} ScoreTransform: 'none' NumObservations: 351 NumTrained: 47 Method: 'LogitBoost' LearnerNames: {'Tree'} ReasonForTermination: 'Terminated normally after completing the requested number of training cycles.' FitInfo: [47×1 double] FitInfoDescription: {2×1 cell} Properties, Methods
MdlFinal
is a ClassificationEnsemble
. To predict whether a radar return is good given predictor data, you can pass the predictor data and MdlFinal
to predict
.
Instead of searching optimal values manually by using the cross-validation option ('KFold'
) and the kfoldLoss
function, you can use the 'OptimizeHyperparameters'
name-value pair argument. When you specify 'OptimizeHyperparameters'
, the software finds optimal parameters automatically using Bayesian optimization. The optimal values obtained by using 'OptimizeHyperparameters'
can be different from those obtained using manual search.
mdl = fitcensemble(X,Y,'OptimizeHyperparameters',{'NumLearningCycles','LearnRate','MaxNumSplits'})
|====================================================================================================================| | Iter | Eval | Objective | Objective | BestSoFar | BestSoFar | NumLearningC-| LearnRate | MaxNumSplits | | | result | | runtime | (observed) | (estim.) | ycles | | | |====================================================================================================================| | 1 | Best | 0.17379 | 6.2592 | 0.17379 | 0.17379 | 137 | 0.001364 | 3 |
| 2 | Accept | 0.17379 | 0.79961 | 0.17379 | 0.17379 | 15 | 0.013089 | 144 |
| 3 | Best | 0.065527 | 1.4585 | 0.065527 | 0.065538 | 31 | 0.47201 | 2 |
| 4 | Accept | 0.074074 | 13.988 | 0.065527 | 0.065549 | 340 | 0.92167 | 7 |
| 5 | Accept | 0.088319 | 0.92718 | 0.065527 | 0.072102 | 22 | 0.2432 | 1 |
| 6 | Accept | 0.074074 | 0.44748 | 0.065527 | 0.071237 | 10 | 0.7177 | 48 |
| 7 | Accept | 0.08547 | 0.52207 | 0.065527 | 0.074847 | 10 | 0.57238 | 2 |
| 8 | Accept | 0.074074 | 0.59154 | 0.065527 | 0.065556 | 11 | 0.97207 | 3 |
| 9 | Best | 0.059829 | 1.6809 | 0.059829 | 0.059648 | 42 | 0.92135 | 343 |
| 10 | Best | 0.054131 | 2.2481 | 0.054131 | 0.054148 | 49 | 0.97807 | 37 |
| 11 | Accept | 0.065527 | 2.1686 | 0.054131 | 0.059479 | 48 | 0.9996 | 2 |
| 12 | Accept | 0.068376 | 2.5909 | 0.054131 | 0.061923 | 58 | 0.91401 | 323 |
| 13 | Accept | 0.17379 | 0.48621 | 0.054131 | 0.062113 | 10 | 0.0010045 | 4 |
| 14 | Accept | 0.17379 | 0.55949 | 0.054131 | 0.059231 | 10 | 0.059072 | 148 |
| 15 | Accept | 0.065527 | 1.9568 | 0.054131 | 0.062559 | 46 | 0.76657 | 19 |
| 16 | Accept | 0.065527 | 2.5692 | 0.054131 | 0.062807 | 57 | 0.64443 | 311 |
| 17 | Accept | 0.17379 | 0.55723 | 0.054131 | 0.062748 | 10 | 0.0035012 | 2 |
| 18 | Accept | 0.12821 | 1.9669 | 0.054131 | 0.062043 | 47 | 0.055757 | 197 |
| 19 | Accept | 0.05698 | 1.2814 | 0.054131 | 0.060837 | 27 | 0.98997 | 12 |
| 20 | Accept | 0.059829 | 1.1975 | 0.054131 | 0.060881 | 26 | 0.99112 | 13 |
|====================================================================================================================| | Iter | Eval | Objective | Objective | BestSoFar | BestSoFar | NumLearningC-| LearnRate | MaxNumSplits | | | result | | runtime | (observed) | (estim.) | ycles | | | |====================================================================================================================| | 21 | Accept | 0.065527 | 1.2255 | 0.054131 | 0.061441 | 25 | 0.99183 | 9 |
| 22 | Accept | 0.17379 | 1.3748 | 0.054131 | 0.061461 | 29 | 0.0032434 | 344 |
| 23 | Accept | 0.068376 | 3.055 | 0.054131 | 0.061768 | 67 | 0.18672 | 11 |
| 24 | Accept | 0.059829 | 5.0035 | 0.054131 | 0.061785 | 119 | 0.3125 | 1 |
| 25 | Accept | 0.059829 | 7.6141 | 0.054131 | 0.061793 | 176 | 0.25401 | 304 |
| 26 | Accept | 0.059829 | 5.1133 | 0.054131 | 0.05988 | 115 | 0.34331 | 343 |
| 27 | Accept | 0.059829 | 7.4027 | 0.054131 | 0.059895 | 178 | 0.26684 | 13 |
| 28 | Accept | 0.059829 | 5.2506 | 0.054131 | 0.059872 | 118 | 0.32365 | 3 |
| 29 | Accept | 0.062678 | 10.523 | 0.054131 | 0.059871 | 238 | 0.22465 | 1 |
| 30 | Accept | 0.14815 | 0.57384 | 0.054131 | 0.059705 | 10 | 0.15205 | 2 |
__________________________________________________________ Optimization completed. MaxObjectiveEvaluations of 30 reached. Total function evaluations: 30 Total elapsed time: 122.7983 seconds. Total objective function evaluation time: 91.3933 Best observed feasible point: NumLearningCycles LearnRate MaxNumSplits _________________ _________ ____________ 49 0.97807 37 Observed objective function value = 0.054131 Estimated objective function value = 0.062545 Function evaluation time = 2.2481 Best estimated feasible point (according to models): NumLearningCycles LearnRate MaxNumSplits _________________ _________ ____________ 119 0.3125 1 Estimated objective function value = 0.059705 Estimated function evaluation time = 5.1842
mdl = ClassificationEnsemble ResponseName: 'Y' CategoricalPredictors: [] ClassNames: {'b' 'g'} ScoreTransform: 'none' NumObservations: 351 HyperparameterOptimizationResults: [1×1 BayesianOptimization] NumTrained: 119 Method: 'LogitBoost' LearnerNames: {'Tree'} ReasonForTermination: 'Terminated normally after completing the requested number of training cycles.' FitInfo: [119×1 double] FitInfoDescription: {2×1 cell} Properties, Methods
Tbl
— Sample dataSample data used to train the model, specified as a table. Each
row of Tbl
corresponds to one observation, and
each column corresponds to one predictor variable. Tbl
can
contain one additional column for the response variable. Multi-column
variables and cell arrays other than cell arrays of character vectors
are not allowed.
If Tbl
contains the response variable
and you want to use all remaining variables as predictors, then specify
the response variable using ResponseVarName
.
If Tbl
contains the response
variable, and you want to use a subset of the remaining variables
only as predictors, then specify a formula using formula
.
If Tbl
does not contain the response
variable, then specify the response data using Y
.
The length of response variable and the number of rows of Tbl
must
be equal.
Note
To save memory and execution time, supply X
and Y
instead
of Tbl
.
Data Types: table
ResponseVarName
— Response variable nameTbl
Response variable name, specified as the name of the response variable in
Tbl
.
You must specify ResponseVarName
as a character
vector or string scalar. For example, if Tbl.Y
is the
response variable, then specify ResponseVarName
as
'Y'
. Otherwise, fitcensemble
treats all columns of Tbl
as predictor
variables.
The response variable must be a categorical, character, or string array, logical or numeric vector, or cell array of character vectors. If the response variable is a character array, then each element must correspond to one row of the array.
For classification, you can specify the order of the classes using the
ClassNames
name-value pair argument. Otherwise,
fitcensemble
determines the class order, and stores
it in the Mdl.ClassNames
.
Data Types: char
| string
formula
— Explanatory model of response variable and subset of predictor variablesExplanatory model of the response variable and a subset of the predictor variables,
specified as a character vector or string scalar in the form
'Y~X1+X2+X3'
. In this form, Y
represents the
response variable, and X1
, X2
, and
X3
represent the predictor variables.
To specify a subset of variables in Tbl
as predictors for
training the model, use a formula. If you specify a formula, then the software does not
use any variables in Tbl
that do not appear in
formula
.
The variable names in the formula must be both variable names in Tbl
(Tbl.Properties.VariableNames
) and valid MATLAB® identifiers.
You can verify the variable names in Tbl
by using the isvarname
function. The following code returns logical 1
(true
) for each variable that has a valid variable name.
cellfun(@isvarname,Tbl.Properties.VariableNames)
Tbl
are not valid, then convert them by using the
matlab.lang.makeValidName
function.Tbl.Properties.VariableNames = matlab.lang.makeValidName(Tbl.Properties.VariableNames);
Data Types: char
| string
X
— Predictor dataPredictor data, specified as numeric matrix.
Each row corresponds to one observation, and each column corresponds to one predictor variable.
The length of Y
and the number of rows
of X
must be equal.
To specify the names of the predictors in the order of their
appearance in X
, use the PredictorNames
name-value
pair argument.
Data Types: single
| double
Y
— Response dataResponse data, specified as a categorical, character, or string array,
logical or numeric vector, or cell array of character vectors. Each entry in
Y
is the response to or label for the observation in
the corresponding row of X
or Tbl
.
The length of Y
and the number of rows of
X
or Tbl
must be equal. If the
response variable is a character array, then each element must correspond to
one row of the array.
You can specify the order of the classes using the
ClassNames
name-value pair argument. Otherwise,
fitcensemble
determines the class order, and stores
it in the Mdl.ClassNames
.
Data Types: categorical
| char
| string
| logical
| single
| double
| cell
Specify optional
comma-separated pairs of Name,Value
arguments. Name
is
the argument name and Value
is the corresponding value.
Name
must appear inside quotes. You can specify several name and value
pair arguments in any order as
Name1,Value1,...,NameN,ValueN
.
'CrossVal','on','LearnRate',0.05
specifies to implement
10-fold cross-validation and to use 0.05
as the learning
rate.Note
You cannot use any cross-validation name-value pair argument along with the
'OptimizeHyperparameters'
name-value pair argument. You can modify
the cross-validation for 'OptimizeHyperparameters'
only by using the
'HyperparameterOptimizationOptions'
name-value pair
argument.
'Method'
— Ensemble aggregation method'Bag'
| 'Subspace'
| 'AdaBoostM1'
| 'AdaBoostM2'
| 'GentleBoost'
| 'LogitBoost'
| 'LPBoost'
| 'RobustBoost'
| 'RUSBoost'
| 'TotalBoost'
Ensemble aggregation method, specified as the comma-separated pair
consisting of 'Method'
and one of the following
values.
Value | Method | Classification Problem Support | Related Name-Value Pair Arguments |
---|---|---|---|
'Bag' | Bootstrap aggregation (bagging, for example,
random forest[2]) — If
'Method' is
'Bag' , then
fitcensemble uses bagging
with random predictor selections at each split
(random forest) by default. To use bagging without
the random selections, use tree learners whose
'NumVariablesToSample' value is
'all' or use discriminant
analysis learners. | Binary and multiclass | N/A |
'Subspace' | Random subspace | Binary and multiclass | NPredToSample |
'AdaBoostM1' | Adaptive boosting | Binary only | LearnRate |
'AdaBoostM2' | Adaptive boosting | Multiclass only | LearnRate |
'GentleBoost' | Gentle adaptive boosting | Binary only | LearnRate |
'LogitBoost' | Adaptive logistic regression | Binary only | LearnRate |
'LPBoost' | Linear programming boosting — Requires Optimization Toolbox™ | Binary and multiclass | MarginPrecision |
'RobustBoost' | Robust boosting — Requires Optimization Toolbox | Binary only | RobustErrorGoal ,
RobustMarginSigma ,
RobustMaxMargin |
'RUSBoost' | Random undersampling boosting | Binary and multiclass | LearnRate ,
RatioToSmallest |
'TotalBoost' | Totally corrective boosting — Requires Optimization Toolbox | Binary and multiclass | MarginPrecision |
You can specify sampling options
(FResample
, Replace
,
Resample
) for training data when you use
bagging ('Bag'
) or boosting
('TotalBoost'
, 'RUSBoost'
,
'AdaBoostM1'
, 'AdaBoostM2'
,
'GentleBoost'
, 'LogitBoost'
,
'RobustBoost'
, or
'LPBoost'
).
The defaults are:
'LogitBoost'
for binary problems and
'AdaBoostM2'
for multiclass problems
if 'Learners'
includes only tree
learners
'AdaBoostM1'
for binary problems and
'AdaBoostM2'
for multiclass problems
if 'Learners'
includes both tree and
discriminant analysis learners
'Subspace'
if
'Learners'
does not include tree
learners
For details about ensemble aggregation algorithms and examples, see Algorithms, Tips, Ensemble Algorithms, and Choose an Applicable Ensemble Aggregation Method.
Example: 'Method','Bag'
'NumLearningCycles'
— Number of ensemble learning cycles100
(default) | positive integer | 'AllPredictorCombinations'
Number of ensemble learning cycles, specified as the comma-separated
pair consisting of 'NumLearningCycles'
and a positive
integer or 'AllPredictorCombinations'
.
If you specify a positive integer, then, at every learning
cycle, the software trains one weak learner for every template
object in Learners
. Consequently, the
software trains
NumLearningCycles*numel(Learners)
learners.
If you specify 'AllPredictorCombinations'
,
then set Method
to
'Subspace'
and specify one learner only
for Learners
. With these settings, the
software trains learners for all possible combinations of
predictors taken NPredToSample
at a time.
Consequently, the software trains nchoosek
(size(X,2),NPredToSample)
learners.
The software composes the ensemble using all trained learners and
stores them in Mdl.Trained
.
For more details, see Tips.
Example: 'NumLearningCycles',500
Data Types: single
| double
| char
| string
'Learners'
— Weak learners to use in ensemble'discriminant'
| 'knn'
| 'tree'
| weak-learner template object | cell vector of weak-learner template objectsWeak learners to use in the ensemble, specified as the comma-separated
pair consisting of 'Learners'
and a weak-learner
name, weak-learner template object, or cell vector of weak-learner
template objects.
Weak Learner | Weak-Learner Name | Template Object Creation Function | Method
Setting |
---|---|---|---|
Discriminant analysis | 'discriminant' | templateDiscriminant | Recommended for 'Subspace' |
k-nearest neighbors | 'knn' | templateKNN | For 'Subspace' only |
Decision tree | 'tree' | templateTree | All methods except
'Subspace' |
Weak-learner name ('discriminant'
,
'knn'
, or 'tree'
)
— fitcensemble
uses weak learners
created by a template object creation function with default
settings. For example, specifying
'Learners','discriminant'
is the same
as specifying
'Learners',templateDiscriminant()
. See
the template object creation function pages for the default
settings of a weak learner.
Weak-learner template object —
fitcensemble
uses the weak learners
created by a template object creation function. Use the
name-value pair arguments of the template object creation
function to specify the settings of the weak
learners.
Cell vector of m weak-learner template
objects — fitcensemble
grows
m learners per learning cycle (see
NumLearningCycles
). For example,
for an ensemble composed of two types of classification
trees, supply {t1 t2}
, where
t1
and t2
are
classification tree template objects returned by
templateTree
.
The default 'Learners'
value is
'knn'
if 'Method'
is
'Subspace'
.
The default 'Learners'
value is
'tree'
if 'Method'
is
'Bag'
or any boosting method. The default values
of templateTree()
depend on the value of
'Method'
.
For bagged decision trees, the maximum number of decision
splits ('MaxNumSplits'
) is n–1
,
where n
is the number of observations.
The number of predictors to select at random for each split
('NumVariablesToSample'
) is the square root
of the number of predictors. Therefore,
fitcensemble
grows deep decision
trees. You can grow shallower trees to reduce model
complexity or computation time.
For boosted decision trees,
'MaxNumSplits'
is 10 and
'NumVariablesToSample'
is
'all'
. Therefore,
fitcensemble
grows shallow decision
trees. You can grow deeper trees for better accuracy.
See templateTree
for the
default settings of a weak learner. To obtain reproducible results, you
must specify the 'Reproducible'
name-value pair argument of
templateTree
as true
if
'NumVariablesToSample'
is not
'all'
.
For details on the number of learners to train, see
NumLearningCycles
and Tips.
Example: 'Learners',templateTree('MaxNumSplits',5)
'NPrint'
— Printout frequency'off'
(default) | positive integerPrintout frequency, specified as the comma-separated pair consisting
of 'NPrint'
and a positive integer or 'off'
.
To track the number of weak learners or folds that
fitcensemble
trained so far, specify a positive integer. That
is, if you specify the positive integer m:
Without also specifying any cross-validation option
(for example, CrossVal
), then fitcensemble
displays
a message to the command line every time it completes training m weak
learners.
And a cross-validation option, then fitcensemble
displays
a message to the command line every time it finishes training m folds.
If you specify 'off'
, then fitcensemble
does
not display a message when it completes training weak learners.
Tip
When training an ensemble of many weak learners on a large data
set, specify a positive integer for NPrint
.
Example: 'NPrint',5
Data Types: single
| double
| char
| string
'NumBins'
— Number of bins for numeric predictors[]
(empty) (default) | positive integer scalarNumber of bins for numeric predictors, specified as the comma-separated pair
consisting of 'NumBins'
and a positive integer scalar. This argument
is valid only when fitcensemble
uses a tree learner, that is,
'Learners'
is either 'tree'
or a template
object created by using templateTree
.
If the 'NumBins'
value is empty (default), then the software
does not bin any predictors.
If you specify the 'NumBins'
value as a positive integer
scalar, then the software bins every numeric predictor into a specified number of
equiprobable bins, and then grows trees on the bin indices instead of the original data.
If the 'NumBins'
value exceeds the number
(u) of unique values for a predictor, then
fitcensemble
bins the predictor into
u bins.
fitcensemble
does not bin categorical
predictors.
When you use a large training data set, this binning option speeds up training but causes a
potential decrease in accuracy. You can try 'NumBins',50
first, and then
change the 'NumBins'
value depending on the accuracy and training
speed.
A trained model stores the bin edges in the BinEdges
property.
Example: 'NumBins',50
Data Types: single
| double
'CategoricalPredictors'
— Categorical predictors list'all'
Categorical predictors
list, specified as the comma-separated pair consisting of
'CategoricalPredictors'
and one of the values in this table.
Value | Description |
---|---|
Vector of positive integers | Each entry in the vector is an index value corresponding to the column of the
predictor data (X or Tbl ) that contains a
categorical variable. |
Logical vector | A true entry means that the corresponding column of predictor
data (X or Tbl ) is a categorical
variable. |
Character matrix | Each row of the matrix is the name of a predictor variable. The names must match
the entries in PredictorNames . Pad the names with extra blanks so
each row of the character matrix has the same length. |
String array or cell array of character vectors | Each element in the array is the name of a predictor variable. The names must match
the entries in PredictorNames . |
'all' | All predictors are categorical. |
Specification of 'CategoricalPredictors'
is appropriate if:
'Learners'
specifies tree
learners.
'Learners'
specifies
k-nearest learners where all
predictors are categorical.
Each learner identifies and treats categorical predictors in the same way as
the fitting function corresponding to the learner. See 'CategoricalPredictors'
of fitcknn
for k-nearest learners and 'CategoricalPredictors'
of fitctree
for tree learners.
Example: 'CategoricalPredictors','all'
Data Types: single
| double
| logical
| char
| string
| cell
'PredictorNames'
— Predictor variable namesPredictor variable names, specified as the comma-separated pair consisting of
'PredictorNames'
and a string array of unique names or cell array
of unique character vectors. The functionality of 'PredictorNames'
depends on the way you supply the training data.
If you supply X
and Y
, then you
can use 'PredictorNames'
to assign names to the predictor
variables in X
.
The order of the names in PredictorNames
must correspond to the column order of X
.
That is, PredictorNames{1}
is the name of
X(:,1)
,
PredictorNames{2}
is the name of
X(:,2)
, and so on. Also,
size(X,2)
and
numel(PredictorNames)
must be
equal.
By default, PredictorNames
is
{'x1','x2',...}
.
If you supply Tbl
, then you can use
'PredictorNames'
to choose which predictor variables
to use in training. That is, fitcensemble
uses only the
predictor variables in PredictorNames
and the response
variable during training.
PredictorNames
must be a subset of
Tbl.Properties.VariableNames
and cannot
include the name of the response variable.
By default, PredictorNames
contains the
names of all predictor variables.
A good practice is to specify the predictors for training
using either 'PredictorNames'
or
formula
, but not both.
Example: 'PredictorNames',{'SepalLength','SepalWidth','PetalLength','PetalWidth'}
Data Types: string
| cell
'ResponseName'
— Response variable name'Y'
(default) | character vector | string scalarResponse variable name, specified as the comma-separated pair consisting of
'ResponseName'
and a character vector or string scalar.
If you supply Y
, then you can
use 'ResponseName'
to specify a name for the response
variable.
If you supply ResponseVarName
or formula
,
then you cannot use 'ResponseName'
.
Example: 'ResponseName','response'
Data Types: char
| string
'CrossVal'
— Cross-validation flag'off'
(default) | 'on'
Cross-validation flag, specified as the comma-separated pair
consisting of 'Crossval'
and 'on'
or 'off'
.
If you specify 'on'
, then the software implements
10-fold cross-validation.
To override this cross-validation setting, use one of these
name-value pair arguments: CVPartition
, Holdout
, KFold
,
or Leaveout
. To create a cross-validated model,
you can use one cross-validation name-value pair argument at a time
only.
Alternatively, cross-validate later by passing Mdl
to crossval
or crossval
.
Example: 'Crossval','on'
'CVPartition'
— Cross-validation partition[]
(default) | cvpartition
partition objectCross-validation partition, specified as the comma-separated pair consisting of
'CVPartition'
and a cvpartition
partition
object created by cvpartition
. The partition object
specifies the type of cross-validation and the indexing for the training and validation
sets.
To create a cross-validated model, you can use one of these four name-value pair arguments
only: CVPartition
, Holdout
,
KFold
, or Leaveout
.
Example: Suppose you create a random partition for 5-fold cross-validation on 500
observations by using cvp = cvpartition(500,'KFold',5)
. Then, you can
specify the cross-validated model by using
'CVPartition',cvp
.
'Holdout'
— Fraction of data for holdout validationFraction of the data used for holdout validation, specified as the comma-separated pair
consisting of 'Holdout'
and a scalar value in the range (0,1). If you
specify 'Holdout',p
, then the software completes these steps:
Randomly select and reserve p*100
% of the data as
validation data, and train the model using the rest of the data.
Store the compact, trained model in the Trained
property of the cross-validated model.
To create a cross-validated model, you can use one of these
four name-value pair arguments only: CVPartition
, Holdout
, KFold
,
or Leaveout
.
Example: 'Holdout',0.1
Data Types: double
| single
'KFold'
— Number of folds10
(default) | positive integer value greater than 1Number of folds to use in a cross-validated model, specified as the comma-separated pair
consisting of 'KFold'
and a positive integer value greater than 1. If
you specify 'KFold',k
, then the software completes these steps:
Randomly partition the data into k
sets.
For each set, reserve the set as validation data, and train the model
using the other k
– 1 sets.
Store the k
compact, trained models in the cells of a
k
-by-1 cell vector in the Trained
property of the cross-validated model.
To create a cross-validated model, you can use one of these
four name-value pair arguments only: CVPartition
, Holdout
, KFold
,
or Leaveout
.
Example: 'KFold',5
Data Types: single
| double
'Leaveout'
— Leave-one-out cross-validation flag'off'
(default) | 'on'
Leave-one-out cross-validation flag, specified as the comma-separated pair consisting of
'Leaveout'
and 'on'
or
'off'
. If you specify 'Leaveout','on'
, then,
for each of the n observations (where n is the
number of observations excluding missing observations, specified in the
NumObservations
property of the model), the software completes
these steps:
Reserve the observation as validation data, and train the model using the other n – 1 observations.
Store the n compact, trained models in the cells of an
n-by-1 cell vector in the Trained
property of the cross-validated model.
To create a cross-validated model, you can use one of these
four name-value pair arguments only: CVPartition
, Holdout
, KFold
,
or Leaveout
.
Example: 'Leaveout','on'
'ClassNames'
— Names of classes to use for trainingNames of classes to use for training, specified as the comma-separated pair consisting of
'ClassNames'
and a categorical, character, or string array, a
logical or numeric vector, or a cell array of character vectors.
ClassNames
must have the same data type as
Y
.
If ClassNames
is a character array, then each element must correspond to
one row of the array.
Use 'ClassNames'
to:
Order the classes during training.
Specify the order of any input or output argument dimension that
corresponds to the class order. For example, use
'ClassNames'
to specify the order of the dimensions
of Cost
or the column order of classification scores
returned by predict
.
Select a subset of classes for training. For example, suppose that the set
of all distinct class names in Y
is
{'a','b','c'}
. To train the model using observations
from classes 'a'
and 'c'
only, specify
'ClassNames',{'a','c'}
.
The default value for ClassNames
is the set of all distinct class names in
Y
.
Example: 'ClassNames',{'b','g'}
Data Types: categorical
| char
| string
| logical
| single
| double
| cell
'Cost'
— Misclassification costMisclassification cost, specified as the comma-separated pair
consisting of 'Cost'
and a square matrix or structure.
If you specify:
The square matrix Cost
, then Cost(i,j)
is
the cost of classifying a point into class j
if
its true class is i
. That is, the rows correspond
to the true class and the columns correspond to the predicted class.
To specify the class order for the corresponding rows and columns
of Cost
, also specify the ClassNames
name-value
pair argument.
The structure S
, then it must have
two fields:
S.ClassNames
, which contains the
class names as a variable of the same data type as Y
S.ClassificationCosts
, which contains
the cost matrix with rows and columns ordered as in S.ClassNames
The default is ones(
, where K
) -
eye(K
)K
is
the number of distinct classes.
Note
fitcensemble
uses Cost
to
adjust the prior class probabilities specified in Prior
.
Then, fitcensemble
uses the adjusted prior probabilities
for training and resets the cost matrix to its default.
Example: 'Cost',[0 1 2 ; 1 0 2; 2 2 0]
Data Types: double
| single
| struct
'Prior'
— Prior probabilities'empirical'
(default) | 'uniform'
| numeric vector | structure arrayPrior probabilities for each class, specified as the comma-separated
pair consisting of 'Prior'
and a value in this
table.
Value | Description |
---|---|
'empirical' | The class prior probabilities are the class relative frequencies
in Y . |
'uniform' | All class prior probabilities are equal to 1/K, where K is the number of classes. |
numeric vector | Each element is a class prior probability. Order the elements according to
Mdl.ClassNames or specify the order using the
ClassNames name-value pair argument. The
software normalizes the elements such that they sum to
1 . |
structure array | A structure
|
fitcensemble
normalizes
the prior probabilities in Prior
to sum to 1.
Example: struct('ClassNames',{{'setosa','versicolor','virginica'}},'ClassProbs',1:3)
Data Types: char
| string
| double
| single
| struct
'ScoreTransform'
— Score transformation'none'
(default) | 'doublelogit'
| 'invlogit'
| 'ismax'
| 'logit'
| function handle | ...Score transformation, specified as the comma-separated pair consisting of
'ScoreTransform'
and a character vector, string scalar, or
function handle.
This table summarizes the available character vectors and string scalars.
Value | Description |
---|---|
'doublelogit' | 1/(1 + e–2x) |
'invlogit' | log(x / (1 – x)) |
'ismax' | Sets the score for the class with the largest score to 1, and sets the scores for all other classes to 0 |
'logit' | 1/(1 + e–x) |
'none' or 'identity' | x (no transformation) |
'sign' | –1 for x < 0 0 for x = 0 1 for x > 0 |
'symmetric' | 2x – 1 |
'symmetricismax' | Sets the score for the class with the largest score to 1, and sets the scores for all other classes to –1 |
'symmetriclogit' | 2/(1 + e–x) – 1 |
For a MATLAB function or a function you define, use its function handle for the score transform. The function handle must accept a matrix (the original scores) and return a matrix of the same size (the transformed scores).
Example: 'ScoreTransform','logit'
Data Types: char
| string
| function_handle
'Weights'
— Observation weightsTbl
Observation weights, specified as the comma-separated pair consisting
of 'Weights'
and a numeric vector of positive values
or name of a variable in Tbl
. The software weighs
the observations in each row of X
or Tbl
with
the corresponding value in Weights
. The size of Weights
must
equal the number of rows of X
or Tbl
.
If you specify the input data as a table Tbl
, then
Weights
can be the name of a variable in Tbl
that contains a numeric vector. In this case, you must specify
Weights
as a character vector or string scalar. For example, if
the weights vector W
is stored as Tbl.W
, then
specify it as 'W'
. Otherwise, the software treats all columns of
Tbl
, including W
, as predictors or the
response when training the model.
The software normalizes Weights
to sum up
to the value of the prior probability in the respective class.
By default, Weights
is ones(
,
where n
,1)n
is the number of observations in X
or Tbl
.
Data Types: double
| single
| char
| string
'FResample'
— Fraction of training set to resample1
(default) | positive scalar in (0,1]'Replace'
— Flag indicating to sample with replacement'on'
(default) | 'off'
Flag indicating sampling with replacement, specified as the
comma-separated pair consisting of 'Replace'
and 'off'
or 'on'
.
For 'on'
, the software samples
the training observations with replacement.
For 'off'
, the software samples
the training observations without replacement. If you set Resample
to 'on'
,
then the software samples training observations assuming uniform weights.
If you also specify a boosting method, then the software boosts by
reweighting observations.
Unless you set Method
to 'bag'
or
set Resample
to 'on'
, Replace
has
no effect.
Example: 'Replace','off'
'Resample'
— Flag indicating to resample'off'
| 'on'
Flag indicating to resample, specified as the comma-separated
pair consisting of 'Resample'
and 'off'
or 'on'
.
If Method
is a boosting method, then:
'Resample','on'
specifies to sample training
observations using updated weights as the multinomial sampling
probabilities.
'Resample','off'
(default) specifies to reweight
observations at every learning iteration.
If Method
is 'bag'
,
then 'Resample'
must be 'on'
.
The software resamples a fraction of the training observations (see FResample
)
with or without replacement (see Replace
).
If you specify to resample using Resample
, then it is good
practice to resample to entire data set. That is, use the default setting of 1 for
FResample
.
'LearnRate'
— Learning rate for shrinkage1
(default) | numeric scalar in (0,1]Learning rate for shrinkage, specified as the comma-separated pair consisting of a numeric scalar in the interval (0,1].
To train an ensemble using shrinkage, set LearnRate
to a value less than 1
, for example, 0.1
is a popular choice. Training an ensemble using shrinkage requires more learning iterations, but often achieves better accuracy.
Example: 'LearnRate',0.1
Data Types: single
| double
'LearnRate'
— Learning rate for shrinkage1
(default) | numeric scalar in (0,1]Learning rate for shrinkage, specified as the comma-separated pair consisting of a numeric scalar in the interval (0,1].
To train an ensemble using shrinkage, set LearnRate
to a value less than 1
, for example, 0.1
is a popular choice. Training an ensemble using shrinkage requires more learning iterations, but often achieves better accuracy.
Example: 'LearnRate',0.1
Data Types: single
| double
'RatioToSmallest'
— Sampling proportion with respect to lowest-represented classSampling proportion with respect to the lowest-represented class,
specified as the comma-separated pair consisting of 'RatioToSmallest'
and
a numeric scalar or numeric vector of positive values with length
equal to the number of distinct classes in the training data.
Suppose that there are K
classes
in the training data and the lowest-represented class has m
observations
in the training data.
If you specify the positive numeric scalar s
,
then fitcensemble
samples
observations
from each class, that is, it uses the same sampling proportion for
each class. For more details, see Algorithms.s
*m
If you specify the numeric vector [
,
then s1
,s2
,...,sK
]fitcensemble
samples
observations
from class si
*m
i
, i
=
1,...,K. The elements of RatioToSmallest
correspond
to the order of the class names specified using ClassNames
(see Tips).
The default value is ones(
,
which specifies to sample K
,1)m
observations
from each class.
Example: 'RatioToSmallest',[2,1]
Data Types: single
| double
'MarginPrecision'
— Margin precision to control convergence speed0.1
(default) | numeric scalar in [0,1]Margin precision to control convergence speed, specified as
the comma-separated pair consisting of 'MarginPrecision'
and
a numeric scalar in the interval [0,1]. MarginPrecision
affects
the number of boosting iterations required for convergence.
Tip
To train an ensemble using many learners, specify a small value
for MarginPrecision
. For training using a few learners,
specify a large value.
Example: 'MarginPrecision',0.5
Data Types: single
| double
'RobustErrorGoal'
— Target classification error0.1
(default) | nonnegative numeric scalarTarget classification error, specified as the comma-separated
pair consisting of 'RobustErrorGoal'
and a nonnegative
numeric scalar. The upper bound on possible values depends on the
values of RobustMarginSigma
and RobustMaxMargin
.
However, the upper bound cannot exceed 1
.
Tip
For a particular training set, usually there is an optimal range
for RobustErrorGoal
. If you set it too low or too
high, then the software can produce a model with poor classification
accuracy. Try cross-validating to search for the appropriate value.
Example: 'RobustErrorGoal',0.05
Data Types: single
| double
'RobustMarginSigma'
— Classification margin distribution spread0.1
(default) | positive numeric scalarClassification margin distribution spread over the training
data, specified as the comma-separated pair consisting of 'RobustMarginSigma'
and
a positive numeric scalar. Before specifying RobustMarginSigma
,
consult the literature on RobustBoost
, for example, [19].
Example: 'RobustMarginSigma',0.5
Data Types: single
| double
'RobustMaxMargin'
— Maximal classification margin0
(default) | nonnegative numeric scalarMaximal classification margin in the training data, specified
as the comma-separated pair consisting of 'RobustMaxMargin'
and
a nonnegative numeric scalar. The software minimizes the number of
observations in the training data having classification margins below RobustMaxMargin
.
Example: 'RobustMaxMargin',1
Data Types: single
| double
'NPredToSample'
— Number of predictors to sample1
(default) | positive integerNumber of predictors to sample for each random subspace learner,
specified as the comma-separated pair consisting of 'NPredToSample'
and
a positive integer in the interval 1,...,p, where p is
the number of predictor variables (size(X,2)
or size(Tbl,2)
).
Data Types: single
| double
'OptimizeHyperparameters'
— Parameters to optimize'none'
(default) | 'auto'
| 'all'
| string array or cell array of eligible parameter names | vector of optimizableVariable
objectsParameters to optimize, specified as the comma-separated pair
consisting of 'OptimizeHyperparameters'
and one of
the following:
'none'
— Do not optimize.
'auto'
— Use
{'Method','NumLearningCycles','LearnRate'}
along with the default parameters for the specified
Learners
:
Learners
=
'tree'
(default) —
{'MinLeafSize'}
Learners
=
'discriminant'
—
{'Delta','Gamma'}
Learners
=
'knn'
—
{'Distance','NumNeighbors'}
Note
For hyperparameter optimization,
Learners
must be a single argument,
not a string array or cell array.
'all'
— Optimize all eligible
parameters.
String array or cell array of eligible parameter names
Vector of optimizableVariable
objects,
typically the output of hyperparameters
The optimization attempts to minimize the cross-validation loss
(error) for fitcensemble
by varying the parameters.
For information about cross-validation loss (albeit in a different
context), see Classification Loss. To control the
cross-validation type and other aspects of the optimization, use the
HyperparameterOptimizationOptions
name-value
pair.
Note
'OptimizeHyperparameters'
values override any values you set using
other name-value pair arguments. For example, setting
'OptimizeHyperparameters'
to 'auto'
causes the
'auto'
values to apply.
The eligible parameters for fitcensemble
are:
Method
— Depends on the number of
classes.
Two classes — Eligible methods are
'Bag'
,
'GentleBoost'
,
'LogitBoost'
,
'AdaBoostM1'
, and
'RUSBoost'
.
Three or more classes — Eligible methods are
'Bag'
,
'AdaBoostM2'
, and
'RUSBoost'
.
NumLearningCycles
—
fitcensemble
searches among positive
integers, by default log-scaled with range
[10,500]
.
LearnRate
—
fitcensemble
searches among positive
reals, by default log-scaled with range
[1e-3,1]
.
The eligible hyperparameters for the chosen
Learners
:
Learners | Eligible
Hyperparameters Bold = Used By Default | Default Range |
---|---|---|
'discriminant' | Delta | Log-scaled in the range
[1e-6,1e3] |
DiscrimType | 'linear' ,
'quadratic' ,
'diagLinear' ,
'diagQuadratic' ,
'pseudoLinear' , and
'pseudoQuadratic' | |
Gamma | Real values in
[0,1] | |
'knn' | Distance | 'cityblock' ,
'chebychev' ,
'correlation' ,
'cosine' ,
'euclidean' ,
'hamming' ,
'jaccard' ,
'mahalanobis' ,
'minkowski' ,
'seuclidean' , and
'spearman' |
DistanceWeight | 'equal' ,
'inverse' , and
'squaredinverse' | |
Exponent | Positive values in
[0.5,3] | |
NumNeighbors | Positive integer values log-scaled in the
range [1,
max(2,round(NumObservations/2))] | |
Standardize | 'true' and
'false' | |
'tree' | MaxNumSplits | Integers log-scaled in the range
[1,max(2,NumObservations-1)] |
MinLeafSize | Integers log-scaled in the range
[1,max(2,floor(NumObservations/2))] | |
NumVariablesToSample | Integers in the range
[1,max(2,NumPredictors)] | |
SplitCriterion | 'gdi' ,
'deviance' , and
'twoing' |
Alternatively, use hyperparameters
with your chosen Learners
. Note that you must
specify the predictor data and response when creating an
optimizableVariable
object.
load fisheriris params = hyperparameters('fitcensemble',meas,species,'Tree');
To see the eligible and default hyperparameters, examine
params
.
Set nondefault parameters by passing a vector of
optimizableVariable
objects that have nondefault
values. For example,
load fisheriris params = hyperparameters('fitcensemble',meas,species,'Tree'); params(4).Range = [1,30];
Pass params
as the value of
OptimizeHyperparameters
.
By default, iterative display appears at the command line, and
plots appear according to the number of hyperparameters in the optimization. For the
optimization and plots, the objective function is log(1 + cross-validation loss) for regression and the misclassification rate for classification. To control
the iterative display, set the Verbose
field of the
'HyperparameterOptimizationOptions'
name-value pair argument. To
control the plots, set the ShowPlots
field of the
'HyperparameterOptimizationOptions'
name-value pair argument.
For an example, see Optimize Classification Ensemble.
Example: 'OptimizeHyperparameters',{'Method','NumLearningCycles','LearnRate','MinLeafSize','MaxNumSplits'}
'HyperparameterOptimizationOptions'
— Options for optimizationOptions for optimization, specified as the comma-separated pair consisting of
'HyperparameterOptimizationOptions'
and a structure. This
argument modifies the effect of the OptimizeHyperparameters
name-value pair argument. All fields in the structure are optional.
Field Name | Values | Default |
---|---|---|
Optimizer |
| 'bayesopt' |
AcquisitionFunctionName |
Acquisition functions whose names include
| 'expected-improvement-per-second-plus' |
MaxObjectiveEvaluations | Maximum number of objective function evaluations. | 30 for 'bayesopt' or 'randomsearch' , and the entire grid for 'gridsearch' |
MaxTime | Time limit, specified as a positive real. The time limit is in seconds, as measured by | Inf |
NumGridDivisions | For 'gridsearch' , the number of values in each dimension. The value can be
a vector of positive integers giving the number of
values for each dimension, or a scalar that
applies to all dimensions. This field is ignored
for categorical variables. | 10 |
ShowPlots | Logical value indicating whether to show plots. If true , this field plots
the best objective function value against the
iteration number. If there are one or two
optimization parameters, and if
Optimizer is
'bayesopt' , then
ShowPlots also plots a model of
the objective function against the
parameters. | true |
SaveIntermediateResults | Logical value indicating whether to save results when Optimizer is
'bayesopt' . If
true , this field overwrites a
workspace variable named
'BayesoptResults' at each
iteration. The variable is a BayesianOptimization object. | false |
Verbose | Display to the command line.
For details, see the
| 1 |
UseParallel | Logical value indicating whether to run Bayesian optimization in parallel, which requires Parallel Computing Toolbox™. Due to the nonreproducibility of parallel timing, parallel Bayesian optimization does not necessarily yield reproducible results. For details, see Parallel Bayesian Optimization. | false |
Repartition | Logical value indicating whether to repartition the cross-validation at every iteration. If
| false |
Use no more than one of the following three field names. | ||
CVPartition | A cvpartition object, as created by cvpartition . | 'Kfold',5 if you do not specify any cross-validation
field |
Holdout | A scalar in the range (0,1) representing the holdout fraction. | |
Kfold | An integer greater than 1. |
Example: 'HyperparameterOptimizationOptions',struct('MaxObjectiveEvaluations',60)
Data Types: struct
Mdl
— Trained classification ensemble modelClassificationBaggedEnsemble
model object | ClassificationEnsemble
model object | ClassificationPartitionedEnsemble
cross-validated
model objectTrained ensemble model, returned as one of the model objects in this table.
Model Object | Specify Any Cross-Validation Options? | Method
Setting | Resample
Setting |
---|---|---|---|
ClassificationBaggedEnsemble | No | 'Bag' | 'on' |
ClassificationEnsemble | No | Any ensemble aggregation method for classification | 'off' |
ClassificationPartitionedEnsemble | Yes | Any ensemble aggregation method for classification | 'off' or
'on' |
The name-value pair arguments that control cross-validation
are CrossVal
, Holdout
,
KFold
, Leaveout
, and
CVPartition
.
To reference properties of Mdl
, use dot notation. For
example, to access or display the cell vector of weak learner model objects
for an ensemble that has not been cross-validated, enter
Mdl.Trained
at the command line.
NumLearningCycles
can vary from a few dozen to a few
thousand. Usually, an ensemble with good predictive power requires from a few
hundred to a few thousand weak learners. However, you do not have to train an
ensemble for that many cycles at once. You can start by growing a few dozen
learners, inspect the ensemble performance and then, if necessary, train more
weak learners using resume
for classification
problems.
Ensemble performance depends on the ensemble setting and the setting of the weak learners. That is, if you specify weak learners with default parameters, then the ensemble can perform poorly. Therefore, like ensemble settings, it is good practice to adjust the parameters of the weak learners using templates, and to choose values that minimize generalization error.
If you specify to resample using Resample
, then it is
good practice to resample to entire data set. That is, use the default setting
of 1
for FResample
.
If the ensemble aggregation method (Method
) is
'bag'
and:
The misclassification cost (Cost
) is highly
imbalanced, then, for in-bag samples, the software oversamples
unique observations from the class that has a large penalty.
The class prior probabilities (Prior
) are
highly skewed, the software oversamples unique observations from the
class that has a large prior probability.
For smaller sample sizes, these combinations can result in a low
relative frequency of out-of-bag observations from the class that has a large
penalty or prior probability. Consequently, the estimated out-of-bag error is
highly variable and it can be difficult to interpret. To avoid large estimated
out-of-bag error variances, particularly for small sample sizes, set a more
balanced misclassification cost matrix using Cost
or a less
skewed prior probability vector using Prior
.
Because the order of some input and output arguments correspond to the
distinct classes in the training data, it is good practice to specify the class
order using the ClassNames
name-value pair argument.
To determine the class order quickly, remove all observations from
the training data that are unclassified (that is, have a missing
label), obtain and display an array of all the distinct classes, and
then specify the array for ClassNames
. For
example, suppose the response variable (Y
) is a
cell array of labels. This code specifies the class order in the
variable
classNames
.
Ycat = categorical(Y); classNames = categories(Ycat)
categorical
assigns
<undefined>
to unclassified
observations and categories
excludes
<undefined>
from its output. Therefore,
if you use this code for cell arrays of labels or similar code for
categorical arrays, then you do not have to remove observations with
missing labels to obtain a list of the distinct classes.To specify that the class order from lowest-represented label to
most-represented, then quickly determine the class order (as in the
previous bullet), but arrange the classes in the list by frequency
before passing the list to ClassNames
. Following
from the previous example, this code specifies the class order from
lowest- to most-represented in
classNamesLH
.
Ycat = categorical(Y); classNames = categories(Ycat); freq = countcats(Ycat); [~,idx] = sort(freq); classNamesLH = classNames(idx);
After training a model, you can generate C/C++ code that predicts labels for new data. Generating C/C++ code requires MATLAB Coder™. For details, see Introduction to Code Generation.
For details of ensemble aggregation algorithms, see Ensemble Algorithms.
If you set Method
to be a boosting algorithm and
Learners
to be decision trees, then the software grows
shallow decision trees by default. You can adjust tree depth by specifying the
MaxNumSplits
, MinLeafSize
, and
MinParentSize
name-value pair arguments using templateTree
.
For bagging ('Method','Bag'
),
fitcensemble
generates in-bag samples by oversampling
classes with large misclassification costs and undersampling classes with small
misclassification costs. Consequently, out-of-bag samples have fewer
observations from classes with large misclassification costs and more
observations from classes with small misclassification costs. If you train a
classification ensemble using a small data set and a highly skewed cost matrix,
then the number of out-of-bag observations per class can be low. Therefore, the
estimated out-of-bag error can have a large variance and can be difficult to
interpret. The same phenomenon can occur for classes with large prior
probabilities.
For the RUSBoost ensemble aggregation method
('Method','RUSBoost'
), the name-value pair argument
RatioToSmallest
specifies the sampling proportion for
each class with respect to the lowest-represented class. For example, suppose
that there are two classes in the training data: A and
B. A has 100 observations and
B has 10 observations. Suppose also that the
lowest-represented class has m
observations in the
training data.
If you set 'RatioToSmallest',2
, then
= s
*m
2*10
= 20
. Consequently,
fitcensemble
trains every learner using 20
observations from class A and 20 observations from
class B. If you set 'RatioToSmallest',[2
2]
, then you obtain the same result.
If you set 'RatioToSmallest',[2,1]
, then
= s1
*m
2*10
= 20
and
= s2
*m
1*10
= 10
. Consequently,
fitcensemble
trains every learner using 20
observations from class A and 10 observations from
class B.
For dual-core systems and above, fitcensemble
parallelizes
training using Intel® Threading Building Blocks (TBB). For details on Intel TBB, see https://software.intel.com/en-us/intel-tbb.
[1] Breiman, L. “Bagging Predictors.” Machine Learning. Vol. 26, pp. 123–140, 1996.
[2] Breiman, L. “Random Forests.” Machine Learning. Vol. 45, pp. 5–32, 2001.
[3] Freund, Y. “A more robust boosting algorithm.” arXiv:0905.2138v1, 2009.
[4] Freund, Y. and R. E. Schapire. “A Decision-Theoretic Generalization of On-Line Learning and an Application to Boosting.” J. of Computer and System Sciences, Vol. 55, pp. 119–139, 1997.
[5] Friedman, J. “Greedy function approximation: A gradient boosting machine.” Annals of Statistics, Vol. 29, No. 5, pp. 1189–1232, 2001.
[6] Friedman, J., T. Hastie, and R. Tibshirani. “Additive logistic regression: A statistical view of boosting.” Annals of Statistics, Vol. 28, No. 2, pp. 337–407, 2000.
[7] Hastie, T., R. Tibshirani, and J. Friedman. The Elements of Statistical Learning section edition, Springer, New York, 2008.
[8] Ho, T. K. “The random subspace method for constructing decision forests.” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 20, No. 8, pp. 832–844, 1998.
[9] Schapire, R. E., Y. Freund, P. Bartlett, and W.S. Lee. “Boosting the margin: A new explanation for the effectiveness of voting methods.” Annals of Statistics, Vol. 26, No. 5, pp. 1651–1686, 1998.
[10] Seiffert, C., T. Khoshgoftaar, J. Hulse, and A. Napolitano. “RUSBoost: Improving classification performance when training data is skewed.” 19th International Conference on Pattern Recognition, pp. 1–4, 2008.
[11] Warmuth, M., J. Liao, and G. Ratsch. “Totally corrective boosting algorithms that maximize the margin.” Proc. 23rd Int’l. Conf. on Machine Learning, ACM, New York, pp. 1001–1008, 2006.
To perform parallel hyperparameter optimization, use the
'HyperparameterOptimizationOptions', struct('UseParallel',true)
name-value pair argument in the call to this function.
For more information on parallel hyperparameter optimization, see Parallel Bayesian Optimization.
For more general information about parallel computing, see Run MATLAB Functions with Automatic Parallel Support (Parallel Computing Toolbox).
ClassificationBaggedEnsemble
| ClassificationEnsemble
| ClassificationPartitionedEnsemble
| predict
| templateDiscriminant
| templateKNN
| templateTree
A modified version of this example exists on your system. Do you want to open this version instead?
You clicked a link that corresponds to this MATLAB command:
Run the command by entering it in the MATLAB Command Window. Web browsers do not support MATLAB commands.
Choose a web site to get translated content where available and see local events and offers. Based on your location, we recommend that you select: .
Select web siteYou can also select a web site from the following list:
Select the China site (in Chinese or English) for best site performance. Other MathWorks country sites are not optimized for visits from your location.