MATLAB Classification Learner App

"When choosing a Holdout validation method before training any of the available models so that you can compare all the models in your session using the same validation technique", does the Classification Learner train all models on the same training data samples and compute validation accuracy on the same validation data samples, or do the data samples differ each time a new model is trained?

[Image Analyst]

Not sure but I would expect all models would use the same training observation data with the same holdouts, otherwise comparing them would not be very accurate, especially for data sets with not many observations. That is for a single training of all models. Of course if you do a training of all the models again, different holdouts may be chosen since they're chosen randomly each time you do a training (different holdouts from run to run).

You might be able to open the source code for it to check, or ask tech support for a definitive answer (if it's not stated in the documentation).

[M. A.hsh]

Thanks for your reply. I need to check if the training and validation of multiple models are performed using the same samples of data for one session.

Answer 1

[Image Analyst]

Here is what the AI Chat Playground says:

Question: Does ClassificationLearner app use the same training and holdout sets for all models?

Answer (sic):

The Classification Learner app in MATLAB does not use the same training and holdout sets for all models. By default, it employs cross-validation to protect against overfitting, but youdata: also choose holdout validation.

When you train a model, the app creates two models: a validation model (using the selected validation scheme) and a full model (trained on the entire dataset excluding any reserved test data). The validation model's results are displayed for evaluation, while the full model is what gets exported for making predictions.

Here’s a concise example of how to set up a training session in the Classification Learner app:

% Load sample data
load fisheriris

% Open Classification Learner app
classificationLearner

% In the app, select 'New Session' and choose 'From Workspace'
% Select 'meas' as predictors and 'species' as the response variable
% Choose validation scheme (cross-validation or holdout)

This workflow allows you to compare models effectively while ensuring that the training anddata: processes are distinct.{"event":"completed","references":[{"title":"Train Classification Models in Classifi

End of AI Answer

So if you want to use all the same data sets you'll have to do it manually one model at a time programmatically, so that you have the sets stored and they are not chosen independently each time. That said, the AI Chat Playground does get stuff wrong quite often so you might want to call tech support and check before coding it up on your own.