What is the difference between k-fold optimization and k-fold cross-validation in fitcecoc? Isn't it redundant?

1 visualizzazione (ultimi 30 giorni)
I try to classify physiological signals into 3 classes by using the fitcecoc routine in Matlab R2016b. I have configured the function in order to automatically optimize the hyperparameters of the trained SVM. It seems to work pretty fine. However, I have experienced some difficulties in understanding how this magical tool works.
It is claimed in Matlab help that the optimization is performed through a k-fold cross-validation process. I think I understand this part: 3 SVMs (in my case) are trained k times on (k-1)/k of the dataset and tested on the remaining 1/k each one providing a binary loss, in fine, the best hyperparameters are chosen in order to minimize a loss-weighted function.
So, could we say that fitcecoc returns an optimized ECOC SVM object that has been cross-validated? (afterall, it performs a k-fold cross-validation within its optimization algorithm)
It is not clear for me since one can also specify cross-validation input arguments to fitcecoc (that are specifically used for cross-validation) and which have the same effect as both crossval function and method.
Is there any interest to perofm a classical cross-validation after fitcecoc?
Many thanks in advance for enlightening me!

Risposta accettata

Don Mathis
Don Mathis il 8 Set 2017
I think it's fair to say that the returned model has been cross-validated, in the sense that it was chosen based on its cross-validated loss (not its in-sample loss). There should be some slightly optimistic bias because the final hyperparameters chosen were chosen because they had low cross-validation loss, which could be due to chance to some extent. But I would expect that bias to be very small. Cross-validation really just gives you an estimate of the out-of-sample loss of the model, so if you want that estimate for your final model, it makes sense to cross-validate the model one more time after running fitcecoc, preferably using large k (e.g., 10 or more).

Più risposte (0)

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by